Mathematics Items Research Articles

Reliability and Validity of PIRLS and TIMSS

Abstract. Academic achievements are often assessed in written exams and tests using selection-type (e.g., multiple-choice, MC) and supply-type (e.g., constructed-response, CR) item response formats. The present article examines how MC items and CR items differ with regard to reliability and criterion validity in two educational large-scale assessments with 4th-graders. The reading items of PIRLS 2006 were compiled into MC scales, CR scales, and mixed scales. Scale reliabilities were estimated according to item response theory (international PIRLS sample; n = 119,413). MC showed smaller standard errors than CR around the reading proficiency mean, whereas CR was more reliable for low and high proficiency levels. In the German sample (n = 7,581), there was no format-specific differential validity (criterion: German grades, r ≈ .5; Δr = 0.01). The mathematics items of TIMSS 2007 (n = 160,922) showed similar reliability patterns. MC validity was slightly larger than CR validity (criterion: mathematics grades; n = 5,111; r ≈ .5, Δr = −0.02). Effects of format-specific test extensions were very small in both studies. It seems that in PIRLS and TIMSS, reliability and validity do not depend substantially on response formats. Consequently, other response format characteristics (like the cost of development, administration, and scoring) should be considered when choosing between MC and CR.

A quantitative methodology for analyzing the impact of the formulation of a mathematical item on students learning assessment

In this paper, we present a methodological approach to the investigation of the impacts of text formulation on students’ answers in mathematical problem solving-based assessment. After a review of the related literature in Mathematics education and a review of the methodologies used until now to investigate this research issue, we describe in depth our quantitative approach, providing motivations and examples of its statistical relevance and its potentiality in making interesting phenomena emerge, to be interpreted with further qualitative methods. We observed statistically significant evidences of different impacts of the variations on different categories of students (males/females; students with high and low performances in the whole test). The methodology and our preliminary results can inform researchers in mathematics education, teachers and experts in the agencies that are responsible for large-scale students learning assessment in several contexts (national and international).

Italian students’ results in the PISA mathematics test: does reading competence matter?

In Italy, from the 2003 reports to the present, the National Institute for the Educational Evaluation of Instruction and Training (INVALSI) has conducted research on Programme for International Student Assessment (PISA) results in order to understand Italian students’ low achievement in mathematics. In the present paper, data from a representative sample of 15-year-old Italian students who participated in PISA 2012 were analysed. This study’s primary aim is to verify how students’ linguistic competences are associated with their performance in mathematics. For the evaluation of the impact of item reading demand on students’ performance, we selected 24 mathematics items with a high reading demand and 31 mathematics items with a low reading demand, as classified by Italian language and methodology experts. Repeated measure variance analyses were conducted. The results showed differences in function of gender: females are advantaged in mathematics items with a high reading demand, independent of their level of reading literacy. In contrast, males are advantaged in mathematics items with a low reading demand, independent of their level of reading literacy. Possible policy implications are discussed.

Usability of American Sign Language Videos for Presenting Mathematics Assessment Content.

There is considerable interest in determining whether high-quality American Sign Language videos can be used as an accommodation in tests of mathematics at both K-12 and postsecondary levels; and in learning more about the usability (e.g., comprehensibility) of ASL videos with two different types of signers - avatar (animated figure) and human. The researchers describe the results of administering each of nine pre-college mathematics items in both avatar and human versions to each of 31 Deaf participants with high school and post-high school backgrounds. This study differed from earlier studies by obliging the participants to rely on the ASL videos to answer the items. While participants preferred the human version over the avatar version (apparently due largely to the better expressiveness and fluency of the human), there was no discernible relationship between mathematics performance and signed version.

Agreement of Teachers on Evaluating Assessments of Learning Progressions in English Language Arts and Mathematics

AbstractIn this study, we explored the extent to which teachers agree on the ordering and separation of levels of two different learning progressions (LPs) in English language arts (ELA) and mathematics. In a panel meeting akin to a standard‐setting procedure, we asked teachers to link the items and responses of summative educational assessments to LP levels. We evaluated four types of agreement among teachers, with test developers, and with trained raters. Although the results were quite mixed, agreement on the LP levels of mathematics items was generally better than that for ELA tasks. The agreement on the LP levels of ELA sample responses was generally better than that for mathematics. Implications of the results are discussed.

인지진단모형에 의한 TIMSS 2015 수학 문항의 성별 차별기능문항 탐색

인지진단모형에 의한 TIMSS 2015 수학 문항의 성별 차별기능문항 탐색

Teachers’ reflection on PISA items and why they are so hard for students in Serbia

The study explores how teachers perceive and go about students’ thinking in connection to particular mathematical content and how they frame the notion of applied mathematics in their own classrooms. Teachers’ narratives are built around two released PISA 2012 mathematics items, the ‘Drip rate’ and ‘Climbing Mount Fuji’ (will be referred to as the Fuji item). Teachers show concordance as to the reasons that could make either of the items difficult for students and are able to provide more examples justifying their reasoning for the ‘Fuji’ item. Suggestions linked to making the items more familiar to the students mostly relate to de-contextualization of the items’ content towards a more formal mathematical record. The teachers agree that students need only basic mathematical knowledge, at a level learned during elementary school, in order to solve these problems. Yet, at the same time, many teachers have difficulty clearly verbalising which procedures students are expected to follow to be able to solve the tasks. Disagreement among the teachers is noticeable when labelling the most difficult part(s) of each of the selected items. Mathematics teachers show openness for learning on how to create math problems we examined in this study, but question the purpose and meaning in incorporating more such problems in their own teaching.

Uverenja budućih učitelja i vaspitača o prirodi matematike i o učenju matematike

University studies, among other things, are aimed at enabling primary school teachers and pre-school teachers to support the development of mathematical competences of young learners. Although mathematics content knowledge is a major component of the professional body of knowledge required for teaching mathematics, teachers' professional beliefs on what mathematics is and how mathematics is learned have a significant mediating effect on teachers' success in providing genuine opportunities for learning meaningful mathematics. The research goal of the study conducted at the beginning of the second semester, when students encounter their first mathematics course for teachers, was to analyze the prospective teachers' beliefs on the nature of mathematics and on mathematics learning. The student questionnaire consisted of parts of the questionnaire used in the international study TEDS-M and of a small number of mathematics items designed to verify the answers given by the questionnaire respondents. The results revealed a difference between the self-professed beliefs of the students and the approaches they used to respond to the mathematics items. These findings point to the need for providing specific learning opportunities within initial teacher education to help future teachers in developing coherent mathematical knowledge for teaching and consistent professional beliefs.

The Impact of Homogeneity of Answer Choices on Item Difficulty and Discrimination

This study explored the impact of homogeneity of answer choices on item difficulty and discrimination. Twenty-two matched pairs of elementary and secondary mathematics items were administered to randomly equivalent samples of students. Each item pair comparison was treated as a separate study with the set of effect sizes analyzed using meta-analysis and a moderator analysis. The results show that multiple-choice (MC) items with homogeneous answer choices tend to be easier than MC items with nonhomogeneous answer choices, but the magnitude was related to item content (algebra vs. geometry) and answer choice construction strategy. For algebra items, items with homogeneous answer choices are easier than those with nonhomogeneous answer choices. However, the difficulty of geometry items with homogeneous and nonhomogeneous is not statistically different. Taking into account answer choice construction strategy, the findings showed that items with homogeneous answer choices were easier than items with nonhomogeneous answer choices when different strategy was applied. However, the same construction strategy was applied; thus, the difficulty of items with homogeneous answer choices and nonhomogeneous answer choices was not statistically different. In addition, we found that item discrimination does not significantly change across MC items with homogeneous and nonhomogeneous answer choices.

Mathematical Competency Demands of Assessment Items: a Search for Empirical Evidence

The implementation of mathematical competencies in school curricula requires assessment instruments to be aligned with this new view on mathematical mastery. However, there are concerns over whether existing assessments capture the wide variety of cognitive skills and abilities that constitute mathematical competence. The current study applied an explanatory item response modelling approach to investigate how teacher-rated mathematical competency demands could account for the variation in item difficulty for mathematics items from the Programme for International Student Assessment (PISA) 2012 survey and a Norwegian national grade 10 exam. The results show that the rated competency demands can explain slightly more than and less than half of the variance in item difficulty for the PISA and exam items, respectively. This provides some empirical evidence for the relevance of the mathematical competencies for solving the assessment items. The results also show that for the Norwegian exam, only two of the competencies, Reasoning and argument and Symbols and formalism, appear to influence the difficulty of the items, which questions to what extent the exam items capture the variety of cognitive skills and abilities that constitute mathematical competence. We argue that this type of empirical data from the psychometric modelling should be used to improve assessments and assessment items, as well as to inform and possibly further develop theoretical concepts of mathematical competence.

Differences in mathematics achievement according to opportunity to learn: A 4pL item response theory examination

We examine student performance on mathematics items from the 2012 Program for International Student Assessment. Utilizing a 4 parameter logistic Item Response Theory model, we were particularly interested in estimating the probability of a student who was relatively high achieving incorrectly responding to mathematics items as related to having more or less opportunity to learn as rated by the student. Results indicate that students who were similarly high achieving yet had less opportunity to learn were significantly less likely to correctly respond to an item as compared to those who had more opportunity to learn. When accounting for these item-level differences in OTL in mathematics, socioeconomic status was longer related to academic achievement.

Is Difference in Measurement Outcome between Groups Differential Responding, Bias or Disparity? A Methodology for Detecting Bias and Impact from an Attributional Stance

Measurement bias is a crucial concern for test fairness. Impact (true group difference in the measured scores) is of the ultimate interest in many scientific inquiries. This paper revisits and refines the definitions for bias and impact and articulates a conceptual framework that decouples them from differential item functioning. The conditions for showing bias and impact are articulated and a methodology for empirically detecting them is proposed. The framework and methodology hinges on attributing bias and impact to the studied groups by way of matching on balance scores (e.g., propensity scores estimated by the confounding covariates). A real data demonstration comparing two test-language groups on the mathematics items of TIMSS is provided as a proof of concept and guide for application. In closing, we draw readers’ attention to some caveats and suggestions for adopting this conceptual framework and methodology.

Doing, Teaching, Learning and Thinking About Mathematics – On Becoming a Transformative Teacher

I solved many mathematical problems till today - countless academic problems inside the classroom, and a few pragmatic problems outside. At the beginning of my teaching career, I spent significant time convincing my students that mathematics teaching is an algorithmic problem solving of routine mathematical items to get the correct answers. Afterwards, I slowly took a shift from doing mathematics to teaching mathematics, identifying lots of tricks, tips and techniques. I spent more than a decade to train myself with better techniques to become a better mathematics teacher seeking better achievements of students in written tests. Later on, I engaged myself as a math learner and sought the significance of the methods I employed to teach the mathematical concepts, relation, and logics. I am now at the crossroads of searching better alternatives that help students learn mathematics in a meaningful way. I frequently ask myself why I am teaching mathematics. What does a good mathematics teacher mean? What we do is largely guided by what we believe. Questioning on the widely accepted assumptions, examining the deep-rooted beliefs for the positive shift, and highlighting the epiphanies of my professional life could be very essential on becoming a transformative teacher. In this paper, I portray my narratives as a student and as a mathematics teacher to explain my shift towards becoming a transformative teacher. Through my verisimilitude narratives, I invite readers to examine their beliefs and practices on teaching mathematics, and envisage for better alternatives being aware of their limitations and contexts.

Content analysis of the TEOG mathematics test items based on MONE attainments, TIMSS levels, and renewed Bloom Taxonomy<p>TEOG sınavlarındaki matematik test maddelerinin MEB kazanımlarına, TIMSS seviyelerine ve yenilenen Bloom Taksonomisine göre içerik analizi

The aim of the study is to classify and evaluate the mathematics test items in the Transition System of Secondary Education (TEOG) exam according to the attainment levels of the Secondary School National Mathematics Curriculum, TIMSS levels and Bloom’s renewed taxonomy. The study covers the TEOG math items between the years 2013 and 2016. In data analysis, content analysis, a qualitative research technique, was applied. For the classification of the materials, the acquisition list of the eighth grade mathematics course prepared by MONE, the scheme of TIMSS levels, and Bloom’s Renewed Taxonomy were used. Booklet A was taken as a base for the classification for each exam. According to findings, in the TEOG exams applied between 2013-2016, while the TEOG exam was consistently asking questions in certain examinations from certain acquisitions, it was seen that no questions were asked from some of the acquisitions. When the 260 mathematical items in the TEOG exams classified according to TIMSS levels, of the items 33.8% were at "Level 1", 45% were at "Level 2", 19.6% were at "Level 3" and 1.5% were at "Level 4 ". Parallel with these findings, when TEOG mathematics items classified according to Restructured Bloom Cognitive Domain Taxonomy, of the items17,69% were at "Remembering", 18,46% were at "Understanding", 54,23% were at "Implementation", 5,76% were at "Analysis" and 3.6% were at "Evaluation" levels. The results indicated that the majority of the math items in the TEOG exams were at the “Application level” and the number of items in the "Reasoning level" was limited. Such exams, as TEOG, directing students high school in consistent with their cognitive level, should measure higher order skills, as well. ÖzetÇalışmanın amacı, Temel Eğitimden Ortaöğretime Geçiş Sistemi (TEOG) sınavı matematik test maddelerinin, Milli Eğitim Bakanlığı Ortaokul Matematik Programı kazanımlarına, TIMSS düzeylerine ve Bloom’un Yenilenmiş Bilişsel Alan Taksonomisine göre sınıflandırılıp değerlendirilmesidir. 2013-2016 yılları arasında TEOG sınavı matematik soruları araştırma kapsamında incelenmiştir. Verilerin analizinde nitel araştırma tekniklerinden içerik analizi kullanılmıştır. Maddelerin sınıflandırılması için MEB tarafından hazırlanan sekizinci sınıf matematik dersine ait kazanım listesi, TIMSS sorularının düzeylerini gösteren TIMSS düzeyleri şeması ve Bloom’un Yenilenmiş Bilişsel Alan Taksonomisi kullanılmıştır. Sınıflamada her bir TEOG sınavının matematik temel alanına ait A kitapçıkları temel alınmıştır. Bulgulara göre 2013-2016 yılları arasında uygulanan TEOG sınavlarında belli kazanımlardan her sınavda soru sorulurken, bazı kazanımlardan hiç soru sorulmadığı görülmüştür. TEOG sınavlarında sorulan 260 matematik sorusu TIMSS düzeylerine göre sınıflandırıldığında %33,8'i “Düzey 1”, %45'i “Düzey 2”, %19,6'sı “Düzey 3” ve %1,5 'i “Düzey 4”de yer aldığı görülmüştür. TEOG matematik test maddelerinin; Yeniden Yapılandırılan Bloom Bilişsel Alan Taksonomisine göre dağılımına bakıldığında soruların %17,69’u “Hatırlama”, %18,46'sı “Anlama”, % 54,23'ü “Uygulama”, %5,76'sı “Analiz” ve %3,46'sı “Değerlendirme” bilişsel alan basamaklarında olduğu görülmüştür. Sonuçlara göre TEOG matematik test maddelerinin çoğunluğunun “Uygulama” basamağında olduğu, “Analiz” ve “Değerlendirme” basamaklarındaki madde sayısının sınırlı olduğu görülmüştür. TEOG gibi öğrencilerin bilişsel düzeylerine göre liselere yönlendirilmesinin hedeflendiği bir sınavdaki sorular bu amaca uygun olarak üst düzey bilişsel becerileri de ölçecek şekilde hazırlanmalıdır.

Using Think-Aloud Protocols to Uncover Misconceptions and Improve Developmental Math Instruction: An Exploratory Study

Deficiencies in education continue to escalate around the world. The focus on outcomes assessment has narrowed instructional research and curriculum evaluation to standardized testing in certain subject areas. A prototype for a quantitative literacy assessment instrument was developed with the goal of diagnosing student misconceptions of basic mathematics content and changing instructional practices to undo the misconceptions by applying cognitive psychological theory. Two hundred thirty-eight basic math high school students and 209 remedial community college students in New Jersey and New York were administered the instrument, which had been based on coded data from think-aloud protocols. The instrument asked students to answer 20 basic mathematics items and, in addition, to evaluate four possible solution strategies. For each item, frequencies of selected solution strategies and the association between strategy selection and performance on the 20-question math test are presented as a means for improving instruction. Follow-up research is proposed for determining whether undoing the student misconceptions first before teaching material on a new unit of instruction may yield more positive student outcomes.

Eight Grade Students’ Mathematics Achievement in TIMSS 2011 Cognitive Domains- A Comparison across Four Southeast Asian Countries

This article describes secondary data analyses that explores students’ performance in Trends in Mathematics and Science Study (TIMSS) 2011 mathematics items that assess lower order thinking (LOT) and higher order thinking (HOT) in the four participating Southeast Asian countries (Indonesia, Malaysia, Singapore and Thailand). The objectives are to compare the three cognitive domains of knowing, applying and reasoning for these countries, and to study the students’ performance in LOT and HOT items. The analyses were done both qualitatively using document analyses to investigate the cognitive processes incorporated in the education system, and quantitatively using IDB Analyzer to determine students’ performance. The article reports on detailed students’ performance related to the percentage of students at each benchmark and their percentile achievement. The findings reveal that in Indonesia and Thailand, more students were able to answer HOT items, with higher scores than LOT items. For Malaysia, the general student population were able to answer LOT items, with higher scores when compared to HOT items. In Singapore, students at the bottom 25% obtained higher score for LOT items, while the rest performed better on HOT items. However, a higher percentage of students at the weak, intermediate and advanced categories were more able to answer HOT items. The findings seem to suggest that focusing on problem solving may be inadequate. A strategy worth investigating is complementing the infusion of problem solving skills with making mathematical connections to real life situations through non-routine questions. However, more research is required before suggesting any conclusive pedagogical practice.

Effects of mathematics items' language demands for language minority students: Do they differ between grades?

This study investigates the connection between the academic language demands of mathematics test items and the test performance of monolingual students and language minority students for 2796 third and 26,016 fourth grade students in Germany. We analyzed students' responses to the 126 mathematics test items used in both grades using a generalized linear mixed model and included students' socioeconomic status as a covariate. The results indicate that language minority students are not significantly disadvantaged by items with high academic language demands compared to German monolingual students in the same grade. However, language minority students as well as German monolingual students are more strongly affected by items with high academic language demands in third grade than in fourth grade. Thus, the impact of academic language demands seems to depend on grade level rather than on language minority student status.

Effects of variations in task design on mathematics teachers’ learning experiences: a case of a sorting task

The goal of the study presented in this article was to examine how variations in task design may affect mathematics teachers’ learning experiences. The study focuses on sorting tasks, i.e., learning tasks that require grouping a given set of mathematical items, in as many ways as possible, according to different criteria suggested by the learners. We present an example of a sorting task for which the items to be grouped are related to basic concepts of analytical geometry that are connected to the notion of loci of points. Based on a design experiment of three iterations with practicing secondary school mathematics teachers, we report on intended and enacted objects of learning inherent in three versions of the task. Empirically based suggestions are drawn about design of sorting tasks that potentially evoke desirable learning experiences.

The assessment of mathematical literacy of linguistic minority students: Results of a multi-method investigation

Assessing mathematical literacy of students who have limited proficiency in the language of the test is a critical challenge in mathematics education. Previous research indicates that knowledge and competencies of such students are underestimated. This presents a major validity and fairness problem for assessment. Most efforts addressing fairness and validity issues in assessment of linguistic minority students focus on the test language only. To overcome limitations of single approaches, we examine in this study the interaction between the test language and the student language background by means of multiple methods. Thus, we investigate possible linguistic bias of items flagged as functioning differentially (the result of DIF analyses) by means of (a) two levels of expert analyses and (b) student think-aloud protocols to investigate language effects in published mathematics items from the 2000 to 2003 Programme for International Student Assessment (PISA) administration for students attending French schools in Canada and speaking either French or other languages at home. DIF analyses were conducted to identify items on which students from different home language backgrounds attending French schools achieve differently. The expert panels tended to identify surface characteristics of language that may be responsible for group differences but not for the differential effects detected by differential item functioning (DIF). Student think-aloud protocols in part confirm and in part contradict DIF results, providing insights for the source of the differences. Suggestions are provided for further study.

Examinee Non-Effort on Contextualized and Non-Contextualized Mathematics Items in Large-Scale Assessments

In this study it is investigated to what extent contextualized and non-contextualized mathematics test items have a differential impact on examinee effort. Mixture item response theory (IRT) models are applied to two subsets of items from a national assessment on mathematics in the second grade of the pre-vocational track in secondary education in Flanders. One subset focused on elementary arithmetic and consisted of non-contextualized items. Another subset of contextualized items focused on the application of arithmetic in authentic problem-solving situations. Results indicate that differential performance on the subsets is to a large extent due to test effort. The non-contextualized items appear to be much more susceptible to low examinee effort in low-stakes testing situations. However, subgroups of students can be found with regard to the extent to which they show low effort. One can distinguish a compliant, an underachieving, and a dropout group. Group membership is also linked to relevant background characteristics.