What about Tests Make Students Switch Off? Item‐Level Predictors of Disengagement on the PISA

The OECD Programme for International Student Assessment (PISA), launched by governments of the Organisation for Economic Co-operation and Development (OECD) in 1997, aims at assessing some of the key competencies that contribute to the success of 15-year-old individuals, on a regular basis and within an internationally accepted framework. PISA seeks to provide a basis for policy dialogue and for collaboration in defining and implementing educational goals, in innovative ways that reflect judgements about the skills that are relevant to adult life. PISA defines competence as the ability to successfully meet complex demands in varied contexts through the mobilisation of psychosocial resources, including knowledge and skills, motivation, attitudes, emotions, and other social and behavioural components. Measuring and comparing competencies across languages and cultures is a difficult challenge and is being pursued by PISA progressively. PISA focused its first three assessments on literacy skills, defined as the capacity of young adults to access, manage, integrate and evaluate information, to think imaginatively, to hypothesise and discover, and to communicate their thoughts and ideas effectively. The reasoning behind shifting the emphasis from assessing whether students can reproduce what they have learned towards whether they can extrapolate from what they have learned and apply their competencies in novel situations, derives from the nature of knowledge and skills required in modern life. For example, the tasks that can be solved through simple memorisation or with pre-set algorithms are those that are also easiest to digitise, automatise and offshore, and will thus be less relevant in a modern knowledge society. Since there is no overarching cross-national and cross-cultural agreement on what fundamental competencies 15-year-olds should possess, an international assessment such as PISA can only capture a selection of competencies. Moreover, since various methodological constraints limit the nature of competencies that are currently amenable to large scale assessment, PISA cannot capture the entirety of competencies that will make young people successful. However, the findings presented in this article suggest that the competencies that PISA does assess are highly predictive for the future success of students. In addition, PISA provides policy makers and practitioners with useful tools to improve quality, equity and efficiency in education, by revealing some common characteristics of students, schools and education systems that do well. In a modern world, comparative assessments are an essential tool for educational improvement and research shows that the existence of standardised assessments and examinations is one of the most powerful predictors for the success of an education system. That is not hard to understand, because without such assessments, all students, schools and education systems look the same, it is impossible for teachers and school administrators to detect institutional and systemic strengths and weaknesses, and to support and intervene where expectations are not met. Without reliable and comparable information on learning outcomes, teachers and governments alike rely on input-based incentives and policies that are all too often mirrored in large quality variation between schools as well as a strong dependency between learning success and the socio-economic context of students and schools. Last but not least, it is important to keep in mind that the absence of the reading, mathematical and scientific competencies measured by PISA does not automatically imply the presence of all those important competencies that have not been measured.

This article presents findings from a recent study of the education policy uses and impact of international large-scale assessments, namely the Programme for International Student Assessment (PISA). The paper focuses on two overlapping dimensions of PISA’s education policy use in the context of Spain. These include political dimensions, such as the use of PISA to initiate and justify the 2012−2013 educational reforms; and technical dimensions, namely the use of PISA in the development of national-level indicators used to benchmark progress and guide education and curriculum reform. The study points to the growing dominance of PISA as a powerful policy tool. Findings from the paper add to the body of literature on the different ways in which international assessments are used to guide education policy within national spaces and the role of the OECD as an agent of transnational policy steering.

본 연구에서는 대표적인 국제역량평가인 국제학업성취도평가(Programme for International Student Assessment: 이하 PISA)의 금융소양 영역에 대해 소개함으로써 경제교육 관련자들의 관심을 촉구하고, 이에 대한 우리나라와 주변국의 적극적인 참여를 요청하고 있다. PISA 금융소양 영역은 학생들이 오늘날 사회에서 직면하는 금융 분야의 문제 상황에 대해 자신이 가지고 있는 지식을 어떻게 활용할 수 있는가를 평가하는 것에 초점이 맞추어져있다. 우리나라는 예산의 부담, 시행 상의 어려움, 주요 비교대상국의 부재 등의 문제로 PISA 금융소양 평가에 참여하지 않고 있다. 하지만 PISA 금융소양 평가에 참여함으로써 얻을 수 있는 이득은 적지 않다. PISA 결과에 나타난 국가 내 심층 분석 자료는 교육 정책 개선의 방향을 제시하며, 국가 간 비교 자료는 정책을 구체화시킬 수 있는 아이디어를 제공한다. 또한 경제교육과 정이 지식뿐만 아니라 실제 경제생활과 관련된 문제 해결 능력이나 지식 적용 능력을 길러줄 수 있는 방향으로 개선될 수 있도록 하는 데 도움을 줄 수 있다. 마지막으로 금융교육에 대해 미온적인 일반의 인식을 개선하고 학교 및 학교 밖 금융 및 경제교육을 활성화할 수 있는 계기로 삼을 수 있다는 점에서 의미가 있다.This study introduced the financial literacy assessment in PISA(Programme for International Student Assessment), one of the best international competence assessment, and called for the concerns and interests of the authorities concerned with financial education as well as the participation of Korea and neighboring countries in PISA. The financial literacy assessment in PISA focuses on how to assess the students utilizing their knowledge to address the financial problems they face daily. Korea currently does not participate in this PISA assessment due to several reasons: budget constraints for the payment of participant dividends, class absence, loss of focus due to additional assessment hour, and the absence of comparative countries. Nonetheless, this study believe that the following benefits by introducing and participating in the financial literacy assessment in PISA. The national data analysis of PISA test results can suggest the ways to improve financial education policies, while inter-national analysis will provide us of the more specific ideas to realize those policies. It also can help to reform the national curriculum on financial and economic education to cultivate not only economic knowledge but competencies for economic problem-solving and knowledge application in an actual economic life. Lastly, it can be an opportunity for us to invigorate social as well as school financial education, and to modify the public indifferences toward financial education.

This report provides a systematic review and empirical evidence related to the experiences of middle-income countries and economies participating in the Program for International Student Assessment (PISA), 2000 to 2015. PISA is a triennial survey that aims to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. To date, students representing more than 70 countries and economies have participated in the assessment, including 44 middle-income countries, many of which are developing countries receiving foreign aid. This report provides answers to six important questions about these middle-income countries and their experiences of participating in PISA: what is the extent of developing country participation in PISA and other international learning assessments?; why do these countries join PISA?; what are the financial, technical, and cultural challenges for their participation in PISA?; what impact has participation had on their national assessment capacity?; how have PISA results influenced their national policy discussions?; and what does PISA data tell us about education in these countries and the policies and practices that influence student performance? The findings of this report are being used by the Organization for Economic Co-operation and Development (OECD) to support its efforts to make PISA more relevant to a wider range of countries, and by the World Bank as part of its on-going dialogue with its client countries regarding participation in international large-scale assessments.

The Programme for International Student Assessment (PISA) was designed by the Organisation for Economic Cooperation and Development (OECD) to evaluate the quality, equity, and efficiency of school systems around the world. Specifically, the PISA has assessed 15-year-old students’ reading, mathematics, and science literacy on a 3-year cycle, since 2000. Also, the PISA collects information about how those outcomes are related to key demographic, social, economic, and educational variables. However, the preponderance of reports involving PISA data focus on achievement variables and cross-national comparisons of achievement variables. Challenges in evaluating achievement of students from different cultural and educational settings and data concerning students’ approaches to learning, motivation for learning, and opportunities for learning are rarely reported. A main goal of this themed issue of Teachers College Record (TCR) is to move the conversation about PISA data beyond achievement to also include factors that affect achievement (e.g., SES, home environment, strategy use). Also we asked authors to consider how international assessment data can be used for improving learning and education and what appropriate versus inappropriate inferences can be made from the data. In this introduction, we synthesize the six articles in this issue and themes that cut across them. Also we examine challenges associated with using data from international assessments, like the PISA, to inform education policy and practice within and across countries. We conclude with recommendations for collecting and using data from international assessments to inform research, policy, and teaching and learning.

The Programme for International Student Assessment (PISA) was designed by the Organisation for Economic Cooperation and Development (OECD) to evaluate the quality, equity, and efficiency of school systems around the world. Specifically, the PISA has assessed 15-year-old students’ reading, mathematics, and science literacy on a 3-year cycle, since 2000. Also, the PISA collects information about how those outcomes are related to key demographic, social, economic, and educational variables. However, the preponderance of reports involving PISA data focus on achievement variables and cross-national comparisons of achievement variables. Challenges in evaluating achievement of students from different cultural and educational settings and data concerning students’ approaches to learning, motivation for learning, and opportunities for learning are rarely reported. A main goal of this themed issue of Teachers College Record (TCR) is to move the conversation about PISA data beyond achievement to also include factors that affect achievement (e.g., SES, home environment, strategy use). Also we asked authors to consider how international assessment data can be used for improving learning and education and what appropriate versus inappropriate inferences can be made from the data. In this introduction, we synthesize the six articles in this issue and themes that cut across them. Also we examine challenges associated with using data from international assessments, like the PISA, to inform education policy and practice within and across countries. We conclude with recommendations for collecting and using data from international assessments to inform research, policy, and teaching and learning.

ABSTRACTNew evidence presented in this paper shows that existing accounts of Programme for International Student Assessment (PISA) in developing countries underestimate both progress and inequality because they do not take into account the large proportion of 15–16-year olds who are ineligible for the PISA sampling frame, primarily due to dropout or delay. In such countries, who makes it into PISA (sample eligibility) heavily influences representivity and therefore the interpretation of PISA results. We use PISA in Turkey 2003–2012 as a case study to show how combining measures of access (from the Turkish Demographic and Health Surveys) and learning outcomes (from PISA) can account for changes in the underlying population. We find the percentage of 15–16-year olds reaching Level 2 in PISA is up to twice as large as that reflected in official PISA reports. These findings have strong implications for inter-country and inter-temporal comparisons using PISA, particularly for developing countries.

[Extract] Sam Sellar Bob Lingard David Rutkowski Keita Takayama and The Organisation for Economic Cooperation and Development’s (OECD) Programme for International Student Assessment (PISA) has become arguably the most influential and successful international large-scale assessment (ILSA) and represents a major global investment in test development, administration, data collection and analysis. PISA assesses reading, mathematical and scientific literacy and is conducted with 15-year-olds in approximately seventy countries every three years. This chapter examines different approaches to preparing students to sit for PISA. We compare four national cases at different points in time: (1) Prince Edward Island, Canada in PISA 2003; Japan following PISA 2003; Scotland in PISA 2012; and Norway in PISA 2015. Approaches to preparation across the four cases span a continuum from close adherence to the test administration guidelines to structural reforms that increase alignment between curricula and what PISA assesses. We aim to answer the following question: what can...

The OECD Programme for International Student (PISA) is probably the most prominent signpost for the internationalization of educational policy. The PISA reports on the performance of secondary school students have become an important factor for educational policy-making in the developed world. In some states PISA has spurred more educational reforms than anything before it during the last decades. What is more, PISA succeeded in shifting approach and focus in a most sensitive area of domestic policy touching on social justice and the self-understanding of the citizenry: Because of PISA, policy-making in the field of school education changed from normative, input-oriented reasoning to comparative, empirical, output-oriented analysis. The international plane succeeded in establishing itself as indispensable in a field thus far essentially conceived as domestic.PISA owes its impact on educational policy to a mode of which we call governance by It describes mechanisms which impact on a given policy field by shaping the cognitive framework of policy-making through the collection, processing and dissemination of information. International and supranational institutions more and more often take recourse to by information. This article explores the repercussions of by information for international law in an approach that stresses the publicness of public international law and the role of international institutional law in legally framing global governance. Thus far, these questions have been hardly explored, in spite of the enormous impact of PISA on national policy. We hypothesize that this is because the knowledge and experience of international lawyers relate mostly to international treaties and other binding legal instruments. Governance by information, by contrast, determines society indirectly through which establish or contribute to the cognitive setting within which policy-makers operate. But since no legal obligations are imposed upon states or individuals, it escapes the established perspective of international lawyers, just as many other instruments, actors and processes of global do.In the article, we first provide an overview of PISA and its legal framework (II). Thereafter, we explain why PISA should be considered an exercise of public authority and why it therefore needs a solid public law framework. This part develops a concept of international public authority that focuses on the social relevance of official acts and their impact on individual freedom (III). Third, we explore on a theoretical level how a legal framework could be established for new forms of public authority. In the tradition of German and Italian public law scholarship we suggest the doctrinal construction of instruments (Handlungsformen). This doctrinal construction does not rest on the belief that legal concepts automatically evoke legitimacy. Rather, it stresses the communicative function of legal doctrine, which provides a stable basis for the exercise of authority while at the same time serving as a forum for contestation (IV). In the following part, we construct and propose a standard instrument called Policy Assessment (Politikbewertung) which is designed to grasp the thrust of PISA and similar policies in a legally significant manner (V). Subsequently, the legal regime of National Policy is developed by identifying basic legal elements within the legal framework of PISA which we deem instrumental for the legitimacy and effective functioning of this standard instrument. Those elements relate to the mandate on which National Policy needs to be based, the respect for scientific standards and the representativeness of expertise, access to the assessment data, and national ownership of the assessment results. After a critical appraisal of these elements, we consider the repercussions of the legal regime of National Policy thus established for other international institutions venturing in the area of education. The example of PISA demonstrates that by information is based on a quite elaborate legal framework. Standard are a useful doctrinal category for abstracting basic legal elements and principles from this legal framework, which adds to the legitimacy and effectiveness of the authority thus exercised. This abstraction also enables criticism from various theoretical vantage points. Moreover, the establishment of standard gives policy-makers in international institutions a resource for transposing this type of to other issue areas. Once such a legal regime is sufficiently complex to ensure legitimacy and efficiency, it might be applied to other areas. In this respect, doctrinal conceptualization has a rationalizing and clarifying effect.Lastly, we see National Policy as a valuable tool for holding national governments accountable for their performance. As performance has an impact on the legitimacy of public authority, it would be worthwhile to consider to what extent national governments might be obliged de lege ferenda to expose themselves to such accountability mechanisms. As states are less and less able to meet the needs of a globalized world, and as their citizens and economies must compete on worldwide markets, the legitimacy of the state-based structure of the international legal order might fade if states do not continue to perform on a high level.

More than 20 years after the first round of the OECD’s Programme for International Student Assessment (PISA), it has become one of the most important large-scale international assessments, at a global level and in particular in Europe. Thus, a growing number of scholars have examined or discussed its research design. Nonetheless, a key feature of PISA seems to have been less examined by critical works: the discourse of the research in the PISA literature. Indeed, the PISA reports appear to be scientific literature, both in textual form and in terms of the intentions expressed. Based on the reports published by the OECD since 1999, this article examines, in line with the French tradition of discourse analysis, the characteristics of research writing in PISA and the explanations of its conceptual approach, test design or result analysis. This analysis shows the differences between the research discourse in PISA and typical scientific rhetoric, particularly in terms of constructing a contradictory theoretical dialogue or a discussion of the limits of its approach. Finally, the article questions how these differences indicate a particular construction of education research within PISA and its possible relationship with the global or European dynamics of this scientific field.

Participation in the Organization for Economic Co-operation and Development's (OECD) Program for International Student Assessment (PISA) has continuously expanded: from 43 systems in 2000 to 65 systems in the 2012 cycle, with 71 signed up for PISA 2015. There also has been a growth in sub-national participation, expanding PISA's reach beyond the nation-state. This paper explores sub-national PISA participation in Canada and the USA, asking how PISA is being used within sub-national education policy spaces. We draw on analysis of documents and data from interviews with officials at sub-national, national, and international levels. Findings illustrate some of the diverse motivations and uses of PISA, providing insights into the effects of PISA at the sub-national scale. As such, we argue that competitive comparison in education has deepened through the enhanced granularity of international large-scale assessment data to new scales beyond the nation-state.

This article advances the enlightened discussion of the nature, logic, and possible effects of the Programme for International Student Assessment (PISA). The purpose is to analyse the assumptions regarding how PISA is to achieve its intended effects, that is, to reconstruct PISA's programme theory (PT) and to probe the validity of its underlying assumptions. The article demonstrates that PISA's PT has low internal validity. However, some PISA assumptions are consistent, for example, the assumption that legitimisation activities justify PISA as a transnational benchmarking system measuring education system performance. PISA exemplifies systemic evaluation governance: all actors in the field are expected to use PISA results to react to and reflect on their own practice, compare themselves with others, and then act accordingly to improve education systems and school practice, though no activities or resources are allocated to change school practice. There is no empirical research into how systemic evaluation governance works in practice that can be used to probe PISA's external validity. PISA's PT is in line with discourse policy, governance theory, and school effectiveness research, but whether and under what conditions and how PISA helps change education systems and school practice are empirical questions waiting to be answered.

We experience the emergence of a global educational reform movement, where the OECD (Organization for Economic Cooperation and Development) through its project PISA (Programme for International Student Assessment) has become the key driver. PISA and its focus on league tables and rankings influence educational debates and educational policy world-wide. The OECD is, with PISA as the main instrument, emerging as a kind of global ministry of education, promoting theirown standardized curriculum and system of quality assessment. PISA is designed to be used by the 30+ modern, highly developed countries in the OECD, but is also used by some 40 less developed non-OECD countries as a benchmark for their education system. This influence of OECD will be further widened by a version of PISA that will target developing countries, “PISA for development”. This instrument has the same underlying assumptions and ideals as PISA: the main concern is the national economy, not the personal development of the learner. There is also the underlying assumption that competition is always good, and that a free-market economy always promotes quality. The increasing role taken by the OECD is pushing aside the influence of international organization with different agendas and ideals, like UNESCO and UNICEF. Since studies like PISA by design cannot identify causal relationships behind neither success nor failure, the educational consequences of the studies are not clear. In many countries, PISA results are used to legitimize market-driven reforms, control of the teachers, payment by test results for teachers and principals, erosion of the public school system, privatization and the introduction of more testing regimes.In this development, the OECD now operates in close contact with the world’s largest commercial company in the education sector, Pearson Inc. The success of PISA as an instrument of governance is currently expanded also to target schools and their teaching in a more direct way: a PISA-like instrument, “PISA for Schools” is developed for local use, for schools and school districts, enabling them to compare their own schools to “PISA winners”. This development may also create anxiety and concern not only at the national or federal level, but also at the local level. This test is also a commercial product, opening up a large and untapped market.

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores between 2003 and 2012. New evidence presented in this paper suggests that existing accounts have underestimated both progress and inequity in Turkey because they did not take into account the large proportion of 15-16 year-olds that are ineligible for the PISA sampling frame, either because they are no longer in school or because they are severely delayed. Using Turkey’s Demographic and Health Survey (DHS) data for 2003, 2008 and 2013 we show that the proportion of 15-16 year-olds that were eligible for the PISA sample in Turkey has nearly doubled from 45% in PISA 2003, to 80% in PISA 2012. By combining DHS data on access/attainment and PISA survey data on learning outcomes we show that: (1) the improvement in the percentage of 15-16 year-olds reaching Level 2 in PISA (functional literacy and functional numeracy) is up to twice as large as that reflected in official PISA reports, (2) the gap in functional literacy rates between rich and poor youth in 2012 is 2.3 times as large as was previously thought, and (3) contrary to earlier research the gap between rich and poor has not declined between 2003 and 2012. The paper emphasises the importance of accounting for sample eligibility and representivity when making inter-country and inter-temporal comparisons using international assessment data, particularly for developing countries with expanding education systems.

This study aims to investigate which factors are associated with Turkey's 15-year-olds' scoring above the OECD average (493) on the PISA'09 reading assessment. Collected from a total of 4,996 15-year-old students from Turkey, data were analyzed by logistic regression analysis in order to model the data of students who were split into two: (1) those students who scored above the OECD average of 493 points on the PISA'09 reading assessment, and (2) those who scored below the average. Specifically, the results indicated that gender, age when first entering school, mother's level of education, the number of books present at home, students' attitudes toward reading, and the amount of time spent learning the test's language (Turkish) per week were those factors found to predict the probability of students' scoring above the OECD average (493) on the PISA'09 reading assessment. The findings of this study contribute to the existing literature within this field by identifying which sources play a role in 15-year-olds' success on the PISA'09 reading assessment in Turkey.Key WordsLogistic Regression, PISA'09, Reading Assessment, Reading Performance, 15-year-olds of Turkey.Worldwide, the effectiveness of education systems has aroused the interest of scholars within the field of education. Most countries receive feedback on how well their education systems prepare students for life (Organization for Economic Cooperation and Development [OECD], 2009a). The Program for International Student Assessment (PISA), triennially organized by the OECD since 2000, has been one of the most significant means which covers reading, mathematics, and science necessary for learners to solve real-life problems and to be prepared for real-life (Cobanoglu & Kasapoglu, 2010). As the major focus of the PISA'09 assessment is on reading literacy (OECD, 2009a), the following section provides an overview of reading literacy.Reading literacy is defined as one's ability to understand, use, reflect on, and engage with written texts. It has been assessed in relation to text format, reading processes (aspects), and situations (OECD, 2009a). In order to assess text format, students' reading literacy is measured by taking into account the range of written material that students will encounter in their later lives, such as forms, application letters, advertisements, among others, by means of both continuous (narration, exposition, and argumentation) and non-continuous texts (lists, forms, graphs, or diagrams). Regarding reading processes, also known as aspects, students are assessed on whether they are able to display proficiency in accessing and retrieving knowledge, in their ability to form a broad understanding of the text, in interpreting it, and in reflecting on its contents, form, and features. Situations, which are defined by the actual use for which a specific text has been constructed, are clustered under four groups: (1) personal (e.g., a novel or a biography); (2) public ( e.g., official documents or announcements); (3) occupational (e.g., a manual or a report); and (4) educational (e.g., a textbook or a worksheet). Since some students may perform better in one of the four groups, it is desirable to include a range of reading situations in the items to be measured (OECD, 2009a).Regarding the results of the PISA'09 reading assessment, Turkey (464) scored below the OECD average (493) and ranked 41st among the 65 participant countries (OECD, 2010). Although Turkey scored higher in the PISA'09 reading assessment than it did in the PISA'03 reading assessment, Turkey has remained in the same level (Level 2) since 2003 (Turkish Foundation of Economic Policy Research, 2010).Factors Affecting Reading Literacy/PerformanceWithin the field of education research, the possible reasons behind or the factors related to reading performance have been studied. A review of related international literature provides a great deal of insight into the factors that affect reading performance. …