Teacher Judgments Research Articles

Context counts: Unveiling the impact of achievement level on teachers’ text assessment

BackgroundTeachers' assessments of student writing proficiency are essential for adaptive instruction and providing feedback. However, studies have revealed a lack of objectivity in text quality ratings of the same text in different contexts, although texts of identical quality should receive identical teacher judgments. AimsOur two studies – one experimental and one more ecologically valid – address whether the achievement level impacts teachers’ individual text assessments even when relevant explicit criteria are used. SamplesIn Study 1, participants were 102 student teachers from a university in Germany. In Study 2, participants were 136 student teachers and 91 experienced teachers from Germany. MethodsIn Study 1, participants were randomly assigned to two groups. Student teachers in the high-achievement level group assessed three high-quality texts and one standard text with average quality. Student teachers in the low-achievement level group assessed three low-quality texts and the same standard text with average quality. Participants in Study 2 assessed ten texts randomly drawn from a large corpus. ResultsThe comparison of the standard text in Study 1 showed that student teachers in the high-level achievement group assessed it more negatively on a holistic scale and three analytic scales compared to the low-level achievement group. Multi-level data analysis in Study 2 revealed a negative influence of the quality of the achievement level on the text assessment for experienced and for student teachers. ConclusionsWe discuss how the achievement level can bias teachers’ judgments of specific student performances when concrete assessment criteria should be used.

Understanding teacher judgments of student motivation: The role of (un-)available cues

BackgroundAccurately judging student motivation enables individualized and student-centered instruction. However, teachers in school tend to judge student motivation inaccurately. Low availability of motivation-related cues, like mastery-approach goals and work-avoidance goals, may explain neglecting these cues in judging motivation. Instead, gender and academic achievement might be overly utilized because they are easily available. AimTo test teachers’ utilization of highly and equally available cues when judging student motivation. MethodsIn the first vignette experiment, pre-service and in-service teachers (N = 205) judged eight fictitious students’ motivation sequentially. Teachers received either achievement goal cues (EG1) or additionally gender and academic achievement cues (EG2), creating an information-adequate environment. In Experiment 2, newly recruited pre-service and in-service teachers (N = 213) evaluated the same vignettes in the same groups, but vignettes were presented simultaneously, and cues had to be memorized, resulting in an information-rich environment. Teachers then formed judgments based solely on their memory without further access to the vignettes. ResultsWhen teachers judged student motivation sequentially, they strongly used mastery-approach goals and work-avoidance goals—regardless of whether other cues were available. In memory-based judgments, teachers primarily used gender and academic achievement. ConclusionsResults demonstrate that in information-rich environments where cues have to be memorized, teachers tend to overlook motivation-relevant cues. Instead, they focus more on cues that do not inherently indicate motivation. These findings suggest that teachers could benefit from assessment environments, like formative assessment, that allow for the direct processing of available cues to better judge student motivation.

Measuring Children’s Developmental Status in China Using the ECDI2030: Comparing with Direct Assessment and Teacher Report

AbstractThis study compared different approaches for monitoring progress towards Sustainable Development Goal Target 4.2, which focuses on the proportion of children who are developmentally on track. UNICEF’s Early Childhood Development Index 2030 (ECDI2030), a parent report measure, was compared with a corresponding direct assessment measure using a sample of children aged 3 and 5 (N = 309; 154 girls) in China at two time points. In the second wave, the study also investigated the correlations and agreement between the ECDI2030 and a teacher report measure for children’s development. Although Cronbach’s alpha indicated that both adult report measures had lower reliability, McDonald’s omega showed comparable reliability among the three measures when the assumption of tau-equivalence was relaxed. Moreover, both adult report measures tended to overestimate children’s developmental levels, and were less effective in capturing the development of older children compared to the direct assessment measure. The correlations between parent report and direct assessment were significant for both girls and boys, urban children, and children from higher socioeconomic quartiles in both waves. Parents’ education levels did not substantially moderate the correlations. Moreover, parent report may not predict children’s development as effectively as direct assessment. Compared to parent report, teacher report was less effective in differentiating children’s development across socioeconomic status and urbanicity. Parent and teacher judgements were more consistent on children’s early learning competencies than on children’s motor and psychosocial skills. Implications of the findings for population-based measurement of early childhood development are discussed.

The effects of special educational needs and socioeconomic status on teachers' and parents' judgements of pupils' cognitive abilities.

Teachers' and parents' judgements of pupils' cognitive abilities influence pupils' daily learning opportunities and experiences, as these judgements affect the difficulty level of materials and instruction that teachers and parents provide. Over time, these judgements thus significantly shape educational success. However, pupils' characteristics, such as special educational needs (SEN), giftedness and socioeconomic status (SES) can influence and bias judgement accuracy. The present study aimed to investigate the relation between pupils' cognitive abilities and their teachers' and parents' judgements of these abilities, and potential bias in these judgements related to SEN, giftedness, and SES. The sample consisted of 1073 primary school pupils from grades 4-6 from 77 classes in 16 schools, and their teachers and parents. Teachers and parents rated their pupils' cognitive abilities. Pupils completed the COVAT-3, a cognitive ability test. Multilevel analyses revealed that parent judgements were significantly higher than teacher judgements, but both informants' judgements were equally strong related to the cognitive ability scores. When controlling for pupils' assessed cognitive abilities, the results revealed small judgement biases: negative for SEN, positive for giftedness, and finally positive for high SES, but only in teachers. Overall, the results indicated that teachers and parents can judge their pupils abilities to a moderate degree, but they also hold judgement biases related to SEN, giftedness and SES. As these biases can affect pupils' opportunities, it is important to increase teachers' and parents' awareness.

Professional Judgment of Teachers in Malaysia Based on Score Modelling Diagnostic

Professional Judgment of Teachers in Malaysia Based on Score Modelling Diagnostic

The Relationship between Human Rights Judgment of Pre-service Primary School Teachers and Taking Human Rights Education Courses at Universities

The Relationship between Human Rights Judgment of Pre-service Primary School Teachers and Taking Human Rights Education Courses at Universities

Almanya'da Çok Kültürlü Ortamda Yetişen Türk Kökenli Çocuklara Türkçe Öğreten Öğretmenlerin Değer Yargıları

The value judgments held by teachers responsible for instructing Turkish to children of Turkish origin in Germany’s multicultural context warrant rigorous analysis. This study seeks to critically examine the role these educators play in perpetuating the Turkish language among new generations, following the mass migration of Turkish citizens to Germany since 1961, and to evaluate the impact of these value judgments on the students’ linguistic and cultural development. The value judgments of teachers are posited as a significant factor in the formation of students’ cultural identities and the preservation of their mother tongue, underscoring the profound influence of educators’ attitudes on students’ language acquisition and cultural integration. In a diverse society like Germany, the preservation and development of cultural identity among Turkish students is imperative for their social integration and personal development. This research aims to contribute to the formulation and enhancement of educational policies for teachers of Turkish abroad by elucidating the effect of teachers’ cultural value judgments on students’ value perceptions. Furthermore, this study aspires to serve as a critical reference for shaping educational strategies for children of Turkish origin in Germany. Keywords: Germany, Turkish, Bilingualism, Value Judgments, Turkish Language Teachers

Development and validation of a text-based instrument measuring teachers’ professional vision of student thinking

Abstract Accurate teacher judgments of student thinking are key to adaptive teaching. Yet, making such judgments represents a particularly challenging task for teachers. Developing instruments that can elicit and capture teachers’ cognitive processes of noticing and interpreting is essential for assessing their skills. These instruments should be sensitive to the teachers’ cognitive processes and specific qualities of student thinking in the respective domain. In this article, we present an approach to develop and validate a text-based vignette test which meets these requirements. We used the example of teachers’ noticing and interpreting of cognitive aspects in learners’ attitudes on creation and evolution. We identified relevant components of student thinking in this domain and included these components systematically varied in written classroom vignettes. Experts confirmed a clear and salient representation of the components of student thinking that we intended to include.

Does teacher judgment accuracy matter? How judgment accuracy, teaching quality, and student achievement development are related

Teacher judgment accuracy is assumed to be positively related to student achievement; however, the empirical evidence for this assumption is inconclusive. Using two accuracy indicators measured at different levels (class/teacher- and student-level), we examined the theoretically hypothesized effects of judgment accuracy on German language achievement over a three-year period, and tested whether teaching quality mediates these effects. We applied multilevel mediation models and small sample methods to data from 35 language teachers and 646 students from 42 classes. While no mediating effects were found, the student-level indicator positively predicted achievement, suggesting that student-level accuracy measures should receive more attention.

A structured discussion of the fairness of GCSE and A level grades in England in summer 2020 and 2021

ABSTRACT In England, examinations for general qualifications (GCSE, AS and A level) were cancelled in summer 2020 and summer 2021 in response to the COVID-19 pandemic and replaced with contingency measures involving teacher judgements. For summer 2020, the intention was to calculate grades using rankings provided by centres, prior attainment data for the current learners and performance data for centres’ candidates in previous years. This approach was later abandoned, and 'centre assessment grades' were issued instead (except where calculated grades were higher). For summer 2021, 'teacher-assessed grades' were awarded to learners. These drastically different to usual approaches to the assessment of high-stakes qualifications raised debate about fairness. This article discusses evidence that can inform judgements about the fairness of these grades, using an existing framework of different ‘senses’ of fairness to provide a structure. In this way, we judged how fair grades were in each sense compared to prior years for each of the approaches used (or nearly used) in summer 2020 and 2021. The analysis highlights tensions in the prioritisation of different aspects of fairness in the unusual circumstances of the pandemic.

Supporting teacher judgement and decision‐making: Using focused analysis to help teachers see students, learning, and quality in assessment data

AbstractThis paper reports results from an Australian study into how teachers see features of quality in student work and connect these to next‐step teaching. Data were drawn from a national 3 year project investigating teacher judgement using A–E standards. The project developed scaled exemplars of authentic student written performance assessments to support teacher judgement and inform next‐step teaching and learning. Fifty‐seven participants created evaluative explanations of grading decisions (i.e. cognitive commentaries), wrote reflective responses and participated in online meeting discussions of their cognitive commentaries. These data were examined using qualitative content and thematic analyses. Findings highlighted how stated standards influence teachers’ judgements of student work. Variation was evident in teachers’ focus on (1) content (i.e. core skills vs. extended thinking), (2) specificity of improvement points and suggested teaching strategies and (3) the connection between identified areas for improvement and teaching strategies. Most teachers were able to identify specific points for improvement. However, in the main, they selected next‐step teaching strategies that were general rather than targeted and specific. Teachers reflected that structured analysis helped direct their thinking and judgements, targeting attention on next‐step teaching. The results suggested that teacher education and professional development should focus on supporting teachers to link specific teaching strategies to identified student weaknesses. This study found that artefacts, such as cognitive commentaries that connect assessment, teaching and learning, can help build professional knowledge and expertise, which remain key components of teacher assessment literacy.

Changing preservice teacher students’ stereotypes and attitudes and reducing judgment biases concerning students of different family backgrounds: Effects of a short intervention

Numerous empirical findings have shown biased judgments of (future) teachers depending on students’ ethnic and social background. Furthermore, research has indicated that (future) teachers’ stereotypes and attitudes differ depending on students’ backgrounds and appear to influence (future) teachers’ judgments. Based on theories of stereotype change, attitude change, and judgment formation, a short intervention was developed to change stereotypes and attitudes and to reduce judgment biases. In an experimental study (within- and between-subject design) with N = 215 preservice teacher students, the effectiveness of the intervention on stereotype change, attitude change, and reduction of judgment distortions was tested. The results showed hypothesized effects of the intervention on stereotypes and attitudes towards students with an immigration background and students with low social status. Furthermore, the intervention showed effects on preservice teacher students’ judgments, especially for low-status students.

Reading and writing difficulties in third- and sixth-grade students: a cross-sectional survey.

In Southern Italy and, specifically, in the region of Campania, many surveys show that the average of students with reading difficulties is much higher than in northern Italy and abroad. On the other hand, specific learning disorders (SLDs) in Campania are much less certified. Since there are no etiological reasons that can explain this apparent inconsistency, an objective of this cross-sectional study was to evaluate the extent of reading/writing difficulties in students from a province of Campania and then to assess the ability of teachers to identify such difficulties in their students. Of a total of 241 enrolled students, 155 (64.31%), including 73 from primary school and 82 from secondary school, belonging to 5 schools in the province of Salerno (Italy), took part in the survey. Students' reading and writing skills were assessed through standardized tests. The tests results were then compared with teacher judgments and context-related variables. At the reading test, 28.7% of primary school and 13.4% of lower secondary school students fell below the 5th percentile for age. Results of the writing test were even more significant: almost half of the students of both levels of education performed below the 5th percentile. Teacher judgments showed higher agreement with standardized assessments in primary (88%, K of Cohen=0.68) than in secondary school (78%, K=0.23). Reading and writing difficulties were common in our sample. While reading skills tended to improve with age, writing difficulties apparently persisted to some extent in third and sixth-grade classes. The accuracy of teacher judgments on reading skills is relatively high, but teachers seem to hardly report reading difficulties "requiring attention." Although less "severe" than others, such difficulties should be considered, mainly because of their potential developmental trajectories.

Teacher judgements and gender achievement gaps in primary education in England, Germany, and the US

We examined whether inaccurate teacher judgements of primary school student achievement correlate with students’ gender and whether such bias contributes to gender achievement gaps in language and mathematics. Our study used ex-post harmonised longitudinal data from England, Germany, and the US. We observed domain-specific teacher judgement bias with a positive bias for girls in the language domain and for boys in mathematics. Furthermore, biased teacher judgements partly mediated the effect of gender on later achievement. Despite these common findings, cross-country differences emerged in the extent of teacher judgement bias as well as its mediation of gender achievement gaps. We conclude that this is a topic of relevance across national contexts and where the institutional and societal setting needs more attention in future research.

Primary school teachers’ judgments of their students’ monitoring and regulation skills

To help students improve their self-monitoring and self-regulation skills, teachers should have an accurate idea of how well students can monitor and regulate their learning. We investigated how accurately primary school teachers can judge their students’ monitoring and regulation accuracy and whether and how student characteristics are related to (the accuracy of) teacher judgments of student monitoring and regulation. Thirty-three teachers, teaching 9––10-year-old students, participated with their classes (N = 495 students). Students completed a multiplication and division task and made monitoring and regulation judgments before and after self-scoring their work. We measured (the accuracy of) teachers’ judgments of their students’ monitoring skills before self-scoring, and of their students’ regulation skills before and after self-scoring. Additionally, we measured teachers’ perceptions of student characteristics (e.g., conscientiousness, general mathematics ability, amount of teacher- student contact). Results showed that the teachers correctly estimated that, in general, their students made quite accurate monitoring and regulation judgments. However, they had difficulties with identifying those students who made substantially inaccurate monitoring and regulation judgments (for whom it is particularly important that the teachers can intervene). When taken together, teachers’ perceptions of student characteristics explained substantial variance in (the accuracy of) teacher judgments of students’ monitoring and regulation skills. Moreover, teacher judgments of students’ monitoring accuracy were more accurate when students were perceived to have learning problems or to be relatively more skilled in mathematics. These findings and measures can ultimately contribute to the design of interventions to help teachers judge and develop their students’ self-regulated learning skills.

Evaluating teacher grading during the pandemic: An analysis of how teacher grades in 2020 compared to exam grades during ‘normal times’

The aim was to evaluate teacher grades awarded in 2020 after GCSE and A-level exams were cancelled due to the pandemic. We identified if there were changes in the relationships between student, school and subject-level features and grades awarded by teacher judgement in 2020 when compared to pre-pandemic years.
 A multi-level modelling analysis was conducted using the GRADE dataset which links data on pupil characteristics with results data and university application data. The model outputs identifying the relationship between student, school and subject characteristics and GCSE and A-level results were compared between 2020 and normal exam years to identify any notable changes.
 A further study focussed on whether university application patterns had an influence on teacher judgements at A-level, since teachers may have known students’ application decisions when deciding on grades. For this analysis university tariff and the degree subject applied to by each student were key variables.
 Analysis showed that across both GCSEs and A-levels there was an average increase in results by around half a grade in 2020. There was also less variation in grades in 2020. However, the majority of relationships between grades and other features studied had not substantially changed.
 Prior attainment was by far the strongest predictor of grades, but this relationship was slightly stronger in 2020 compared to previous years.
 Subjects with more non-exam assessment tended to be those with the largest increase in grades, as well as ‘facilitating’ subjects at A level. There was also a decrease in the attainment gap between male and female students at A level.
 The follow up study indicated that teachers were not influenced by student’s application decisions when allocating grades.
 Findings suggest that teacher grades did not introduce any substantial sources of bias or different patterns of grading in 2020. This research helped to inform policy decisions around national assessment in England in 2021, as well as contributing to the broader understanding of direct grading by teachers.

PROMOTING SELF-EFFICACY OF PROSPECTIVE MUSIC TEACHERS: THEORETICAL INSIGHTS

Scientific literature shows that cultivating self-efficacy is conducive to learner in-depth study of music, independent choices and perseverance in the field of music. However, most current research studies focus on self-efficacy in primary and secondary education, whereas the self-efficacy of prospective music teachers has been scarcely investigated. Considering the importance of self-efficacy in music activities of prospective music teachers, a study on the particularity of prospective music teachers’ self-efficacy has been initiated: the literature analysis method has been used, and a large number of literature sources related to self-efficacy and music education have been analysed. The following problem question is formulated: what is the particularity of self-efficacy in music education? The aim of this research is to analyse the concept of self-efficacy in music education. Research results show that self-efficacy determines the judgment of prospective music teachers on their learning, performance, and teaching abilities. Factors that affect the self-efficacy of prospective music teachers include mastery of experiences, vicarious experience, social persuasion, and emotional and psychological states.

Color Differentiation of Digital Risks in Teacher Education

Introduction. At the present stage of education digital transformation, the urgent task is being addressed to train educators who are ready to meet the challenges and risks of a changing and unstable digital world. Digital risks differentiating and adjusting the course of digital learning for future educators constitute the study problem and goal. The paper draws attention to the issues of digital transformation in the educational system carried out under conditions of uncertainty, as well as ways and opportunities to adjust the course of digital learning, ensuring the readiness of future teachers for the changes that are coming in the near and distant future. Materials and Methods. Leading research methods: system analysis of digital risks in teacher education, scenario modeling of network interactions and digital learning in basic risk situations, pedagogical experiment. The dynamic set of analytical and predictive judgments of future teachers about the possibilities of overcoming risk situations constitute the experimental materials. In research, digital risks are differentiated by three definable parameters: subjective sense of danger, relative risk and probability of threat. Results. The algorithmic model provides risk classification features and color codes. Their influence on the choice of safe behavior strategies in basic risk situations associated with networking and digital learning is described. The study reveals the authorʼs idea of risks color differentiation, which in this paper is illustrated by examples of teachers’ professional activities in networking and digital learning situations. The paper practical significance lies in pedagogical influences of algorithmic model approbation in risks situations of different danger levels: especially dangerous, moderate and underestimated risks in the dynamic digital space. It is concluded that actions in danger situations can be effectively regulated by light signals similar to a modified traffic light. Each signal can be matched with an individual scenario included a stereotypical behavior algorithms set that is assigned considering the real danger and subjective feeling on it, the basic scenario and prediction corresponding to it. The survey final measurements showed that the future teachers who passed the experimental training significantly decreased their subjective sense of danger. Discussion and Conclusion. The pedagogical experiment reveals that the study of risk situations, pedagogical activity scenario modeling and orientation to color cues in typical risk situations reduce the teachers’ uncertainty sense and give positive shifts in teachers’ training.

‘The wisdom of crowds’: When teacher judgments outperform word-frequency as a predictor of students’ vocabulary knowledge

This study investigated the effectiveness of word-frequency and teacher judgments in determining students’ vocabulary knowledge and compared the predictive powers of both approaches when estimating vocabulary knowledge. Twenty-nine second language (L2) Spanish teachers were asked to predict how likely their students would know words from a 216-word Yes/No test that measures knowledge of the first 3,000 words in Spanish. The accuracy of their responses was compared with the results of 1,075 L2 Spanish students who completed the same test. To examine if the results could generalize to other L2 settings, 394 L2 English students completed a 70-word Yes/No test that measures knowledge of the first 14,000 words in English, and 15 L2 English language instructors attempted to predict which words would or would not be recognized. Results showed that for both language contexts, (1) the median teacher rater could assess students’ vocabulary knowledge with an accuracy roughly comparable to frequency, (2) the combination of teachers’ judgments displayed a stronger relationship with students’ performance on the vocabulary test than frequency, since the average of three or more teachers’ ratings improved upon frequency when examined with 1,000 bootstrapped samples, and (3) using teacher judgments and frequency together did not substantially improve the prediction of students’ vocabulary knowledge.

Parent and teacher judgments about children's mathematics and reading competencies in primary school: Do parent judgments associate with children's educational outcomes?

Most research on this topic has focused on examining parent and teacher judgments separately. Therefore, information about the relationship between parent and teacher judgments and children's cognitive competence is relatively sparse. To bridge this research gap, this study aimed to examine the relationships between both parent and teacher judgments and children's mathematics and reading competencies.The present study used data from the Starting Cohort 2 of the German National Educational Panel Study, which primarily collected data on children, but also collected data on parents and educators.The results of multilevel regression models confirmed the significant contribution of both parent and teacher judgments in explaining the variance in mathematics and reading competencies. More importantly, the results indicated that the combined contribution of parent and teacher judgments to children's competencies is more meaningful than their independent judgments. This result highlights the role of family-school partnerships in promoting children's educational success.