Writing Assessment Tasks Research Articles

Assessing the proficiency of large language models in automatic feedback generation: An evaluation study

Assessment feedback is important to student learning. Learning analytics (LA) powered by artificial intelligence exhibits profound potential in helping instructors with the laborious provision of feedback. Inspired by the recent advancements made by Generative Pre-trained Transformer (GPT) models, we conducted a study to examine the extent to which GPT models hold the potential to advance the existing knowledge of LA-supported feedback systems towards improving the efficiency of feedback provision. Therefore, our study explored the ability of two versions of GPT models – i.e., GPT-3.5 (ChatGPT) and GPT-4 – to generate assessment feedback on students' writing assessment tasks, common in higher education, with open-ended topics for a data science-related course. We compared the feedback generated by GPT models (namely GPT-3.5 and GPT-4) with the feedback provided by human instructors in terms of readability, effectiveness (content containing effective feedback components), and reliability (correct assessment on student performance). Results showed that (1) both GPT-3.5 and GPT-4 were able to generate more readable feedback with greater consistency than human instructors, (2) GPT-4 outperformed GPT-3.5 and human instructors in providing feedback containing information about effective feedback dimensions, including feeding-up, feeding-forward, process level, and self-regulation level, and (3) GPT-4 demonstrated higher reliability of feedback compared to GPT-3.5. Based on our findings, we discussed the potential opportunities and challenges of utilising GPT models in assessment feedback generation.

Developing Classroom Assessment Tasks That Evaluate Students’ Abilities to Plan Scientific Investigations

Designing classroom assessment tasks that can gather meaningful assessment information is a complex undertaking. This is especially the case when it comes to developing assessment tasks to evaluate students’ abilities to plan scientific investigations. Such assessment tasks should adequately elicit students’ cognitive understandings related to designing valid and reliable experiments, not just their hands-on manipulative skills. This article proposes a set of guidelines for developing written assessment tasks that can surface students’ cognitive understandings related to planning scientific investigations. We show how the checklist created based on the design guidelines can be used to analyze two classroom assessment tasks. The guidelines will be useful for teachers as they develop new or modify existing classroom assessment tasks to meaningfully evaluate students’ abilities to plan scientific investigations.

Amplifying test-taker voices in the validation of L2 writing assessment tasks

Studies have documented how L2 writers utilize digital tools to mediate the process of writing, yet little is known about test-takers’ perceptions of the utility of digital tools in writing assessment tasks. In a previous investigation (Pusey & Butler, in press), we found that international graduate students’ writing performance was different on assessment tasks that varied in terms of access to external writing resources (e.g., spell-check, dictionaries, internet content searches). To better understand the mechanisms underlying these different outcomes, participants’ (n = 20) perceptions of the two task conditions (with or without access to external resources) were examined based on a questionnaire. Results indicated that participants tended to perceive writing tasks that permit access to external resources as more similar to university writing and as better vehicles for demonstrating their academic writing ability. However, they also perceived this task condition as more difficult than the one which disallowed use of external resources. Regarding enjoyment, test-takers’ perceptions were almost evenly divided. Moreover, additional construct-irrelevant factors (e.g., topic, time, task characteristics and processes) appeared to influence perceptions of the tasks. The findings demonstrate the value of listening to test-taker voices to identify construct-irrelevant factors in writing assessments within the context of an increasingly digitalized world.

Investigating the ecological validity of second language writing assessment tasks

Nowadays, writing in academic settings is mediated by digital technology in essential ways, yet many writing test tasks deny access to the digital resources common to everyday practice (e.g., spell-check, electronic dictionaries). This discrepancy potentially threatens the ecological validity of such tasks and raises questions about construct (under) representation. Informed by an ecological perspective on language and communication (van Lier, 2004), a study was carried out to better understand how access to writing resources impacts writing test task performance. Twenty-two international students enrolled in a graduate program in the United States completed two argumentative writing tasks. On one of the tasks, participants were given full access to writing resources; on the other task, no access to writing resources was given. Average scores on participants' essays were analyzed quantitatively to determine the degree to which, and in what specific domains, performance differed under each condition. In addition, participants’ self-assessments of performance on the tasks were analyzed quantitatively and qualitatively. Results suggest that writers benefitted from having access to writing resources, but the perceived impact of resource accessibility differed from the scores given by raters. These findings have implications for writing test task design and may help inform L2 writing instruction.

Unpacking Writing Assessment Tasks for Junior High School Students: A Comprehensive Analysis

Unpacking Writing Assessment Tasks for Junior High School Students: A Comprehensive Analysis

Copy Skills and Writing Abilities in Children With and Without Specific Learning Disabilities

Copying a text quickly and accurately is important both in school and in daily life. However, this skill has never been systematically studied, either in children with typical development (TD) or in children with specific learning disabilities (SLD). The aim of this research was to study the features of a copy task and its relationship with other writing tasks. For this purpose, 674 children with TD and 65 children with SLD from Grades 6 through 8 in Italy were tested with a copy task and other writing assessment tasks, measuring three aspects of writing: handwriting speed, spelling, and expressive writing. Children with SLD performed worse on the copy task, both in terms of speed and accuracy, than children with TD. Copy speed was predicted by grade level and by all three major writing skills for children with TD but only by handwriting speed and spelling for children with SLD. Copy accuracy was predicted by gender and the three major writing skills for children with TD but only by spelling for children with SLD. These results suggest that children with SLD also have difficulty copying a text and benefit less than children with TD from their other writing skills.

Designing Efficient L2 Writing Assessment Tasks for Low‐Proficiency Learners of English

This report describes the development of efficient second language (L2) writing assessment tasks designed specifically for low‐proficiency learners of English to be included in the TOEFL® Essentials™ test. Based on the can‐do descriptors of the Common European Framework of Reference for Languages for the A1 through B1 levels of proficiency, four task types were identified to be prototypical candidate writing tasks for the target test‐taker population (i.e., adolescent and adult low‐proficiency English learners). Those four task types included: (a) Describe a Photo, (b) Write a Review, (c) Chat With a Friend, and (d) Write an E‐mail. These task types were also considered efficient in the framework of the test in that they had the potential to be accessible to low‐proficiency learners and to elicit sufficient spontaneous writing for assessment purposes within a short period of time. In the current study, eight assessment tasks, two for each task type, were developed and piloted with 169 A1–B1 learners of English from Japan and Colombia. The findings revealed that the Describe a Photo and Write an E‐mail tasks performed the best in eliciting substantial language use and emphasizing distinct performance attributes, both characteristics needed for efficiently measuring test takers' writing proficiency as well as discriminating among proficiency levels at the lower end of the spectrum. The report concludes by highlighting some observations on L2 writing assessment task design for low‐proficiency learners of English.

Use of electronic feedback in second language writing: teachers' and students' views

Feedback has always been considered important in second language writing. Quite recently due to various reasons, electronic feedback has become one of the frequently applied types (Zareekbatani, 2015; Ene & Upton, 2018). The aim of the research study was therefore to identify lecturers’ and students’ views on the use of online comments provided on the second language writing tasks. The data was collected through conducting online semi-structured interviews with undergraduate students and lecturers of one Uzbek university. The findings revealed that a variety of comments given on different aspects of the written assessment tasks in the Google documents and combined with additional oral feedback were effective. The article aims at discussing the detailed findings of the research study and providing possible suggestions for language teachers on the use of electronic feedback in L2 writing.

COVID-19 시대의 국어과 쓰기 평가 실태 조사

The purpose of this study is to identify the condition of non-face-to-face writing assessment and to analyze the problems faced by Korea language teachers to propose an effective non-face-to-face writing assessment method. To this end, 133 Korea language teachers were surveyed on the condition of writing assessment in 2020. The results are as follows. First, Korea language teachers who conducted only face-to-face assessment changed writing assessment task or assessment criteria in consideration of the decrease in the number of face-to-face classes. In particular, in terms of writing assessment tasks, changes such as ‘changing from collaborative writing to individual writing, lowering the level of difficulty in subjects, and reducing the amount of writing’ were notable. Second, Korea language teachers who conducted non-face-to-face assessment performed writing assessment mainly by using non-real-time online platforms or by submitting personal tasks rather than real-time non-face-to-face assessment. This suggests that results-oriented writing assessments were conducted rather than the performance process assessment, and that personalized feedback may not have been faithfully provided. Third, some Korea language teachers who conducted face-to-face assessment did not change their writing assessment task or assessment criteria because of the education authorities’ guidelines to avoid reflecting non-face-to-face writing assessments. And some Korea language teachers who conducted non-face-to-face assessment did not change their writing assessment task or assessment criteria because they performed writing assessment that did not differ from face-to-face assessment using real-time media. Based on these results, we proposed ways to implement effective non-face-to-face write assessment.

Classifying the writing assessment tasks of English as the medium of instruction programs using latent class analysis

This study developed a frame to characterize writing tasks required by English as the medium of instruction (EMI) programs and tested it with 55 writing tasks from two EMI programs (creative arts and environmental studies). Results of latent class analysis (LCA) showed: (1) three classes were salient: response-oriented class, argument-oriented class, and description-oriented class; 2) response-oriented tasks dominated creative arts tasks, argument-oriented tasks dominated environmental studies tasks, whereas description-oriented tasks were used as complementary tasks by both programs; and 3) from foundation stage to advanced stages of study, the use of two dominating classes decreased while the use of the complementary class increased. Implications for analyzing EMI writing tasks and for developing language curriculum are discussed.

Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem.

Student engagement during classes includes behavioural, cognitive and emotional components, and is a pre-requisite for successful active learning environments. A novel approach to measuring student engagement was developed, involving triangulation of real-time student-self report, observation by trained observers and heart rate measurement. The self-report instrument was evaluated in four separate cohorts (n = 123) at Monash University and the University of North Carolina. The six item self-report demonstrated good reliability (Cronbach’s alpha values ranged from 0.7–0.81). The self-report showed predictive validity in that small group activities were rated as significantly more engaging than didactic lecturing. Additionally, there was significant inter-instructor variability and within-class variability, indicating good discrimination between classroom activities. This self-report may prove useful to academic teaching staff in evaluating and refining their active learning activities. Independent observation was not found to correlate with student self-report, due in part to students who were pretending to engage being rated as engaged by an observer. Strikingly, students reported that they were pretending to engage for 23% of class time, even for highly regarded instructors. Individual participants were rated as engaged for 42 of the 46 intervals for which they reported that they had “pretended to engage”, indicating that the two observers were unable to detect disengagement during periods in which students pretended to engage. Instructors should be aware that student cues such as eye contact and nodding may indicate pretending to engage. One particular self-report item; “I tried a new approach or way of thinking about the content”, correlated positively with heart rates, and a controlled study reproduced this finding during two activities that required students to try a new approach to understanding a concept. Agreement with this item also correlated with superior performance on two in-class written assessment tasks (n = 101, p<0.01). Further use of this tool and related educational research may be useful to identify in-class activities that are engaging and likely to lead to improved student attainment of learning outcomes.

The Effects of Task Complexity on Heritage and L2 Spanish Development

Manipulating cognitive demands on second language (L2) tasks, along with the provision of recasts and its effects on L2 development, has motivated recent inquiry within task-based research. However, empirical evidence remains inconclusive as to the impact of task complexity, and it is unknown how it may affect heritage language (HL) development. To address this issue, this study tested 81 adult HL and L2 learners of Spanish. Participants in the experimental conditions completed either a simple or a complex version of a monologic computerized task that delivered written recasts as corrective feedback but differed according to intentional reasoning demands. Participants completed three oral and written assessment tasks to measure development of the Spanish subjunctive in adjectival clauses. Results revealed that the simple group demonstrated greater gains, especially in written production. L2 learners and the HL simple group benefitted more from task-based instruction in comparison to the HL complex group. Findings have implications for the role of prior language experience and task outcomes.

Student performance in high-stakes examinations based on content area in senior secondary (VCE) physical education

ABSTRACTBackground: Senior secondary physical education courses for certification continue to evolve with curricula reform occurring to ensure content is contemporary, student learning outcomes are maximised and assessment practices are valid for determining certification of students. The content of examinable senior secondary physical education courses privilege theoretical concepts over student physical performance of motor skills and this is reflected in the use of written assessment of cognitive outcomes in many courses internationally.Purpose: Student examination data were analysed from the year 12 (exit year) written examination of Victorian Certificate of Education (VCE) Physical Education to determine if student performance varied by Area of Study (content). Additionally, it investigated whether there was a relationship between student performance in each of the four Areas of Study examined and overall examination performance and considered the alignment of curriculum, assessment and pedagogy and the implications the findings may have on the teaching of VCE Physical Education in the future.Methods: A secondary data analysis of student results from the 2011 (n = 9323; M = 5212, F = 4111) and 2012 (n = 8781; M = 5011, F = 3770) VCE Physical Education (Victoria, Australia) examinations were conducted. Examination questions were categorised by content, and means and standard deviations (SD) for discrete and continuous data were calculated, and categorical variables were presented as percentages. Regression analysis was also performed to establish the relationship between student cohort size and examination scores. An independent sample t-test was used to explore the examination scores and each Area of Study scores across 2011 and 2012. A one-way ANOVA were performed to investigate the differences of each Area of Study scores between examination grades from UG to A+.Results: The results showed a positive correlation between VCE Physical Education student cohort size in a school and examination score in 2011 and 2012. Student performance differed across both years (2011 and 2012) and across Areas of Study within each of the years analysed. Students performed significantly lower on questions relating to the ‘planning, implementing and evaluating a training program’ Area of Study in 8 of the 11 possible grades (2011) and 10 of the possible 11 grades (2012) than in each of the other Areas of Study.Discussion and conclusions: This study reveals that student performance on the external VCE Physical Education examination is not consistent across all content areas (Areas of Study). This may suggest that student difficulties in answering questions based on content in ‘planning, implementing and evaluating a training program’ result from topic or content difficulty rather than process or question difficulty. From these findings, implications for teaching examinable physical education effectively include the use of experiential learning and practical experiences to provide students with experiences from which they can draw knowledge when completing written assessment tasks. Additionally, the importance of having the required content knowledge to teach examinable physical education confidently for pre-service and in-service teachers is discussed.

Designing Writing Assessment Tasks for Junior High School

Junior High School is the next step for students in learning English. In elementary school, learning target that should be reached by student is vocabulary mastery, then introduced by four language skills-speaking, reading, listening, and writing- in Junior High School. It can’t be denied, even they have Junior High School, and it is still found students who are difficult to write sentences in English. In fact, when English Education Department (EED) students are asked ‘how to asses students writing for Junior High School?’ They answered by writing paragraph. And, how to give students score? They answered by using rubric like stated at ‘Language assessment book by Arthur Huges’. Actually, writing assessment is not only focused on writing paragraph. For Junior high school, there are some writing performance can be used to assess students writing, and to give score it should be appropriate with the level of the student. The rubric used by EED students is the rubric for English as a second language. Therefore in this paper, writer describes how to design writing assessment task for Junior High School and its rubric. Hopefully, this paper can help EED students in designing writing assessment task when they do teaching practice (PPL) and making instrument for their research.

Writing for transition: The role of food studies in the general academic writing classroom

Consistent with ongoing efforts to improve the transition of students to university, this article conceptualises the general academic writing classroom as a potentially complex but productive pedagogic space. Those who design and deliver curricula in academic writing face particular challenges around translating into practice two principles that are foundational to transitional students’ experience: inclusion and engagement. This article presents food studies as a rich interdisciplinary resource for those who strive to construct an inclusive and stimulating learning environment for transitional students of academic writing. It draws on experience in designing and delivering a first-year unit that develops general academic writing skills and involves scaffolded written assessments tasks culminating in the research-based essay. Because such units are taken by students from a range of disciplines, a specific challenge in setting questions for written assessment tasks is to engage all students in such ways that learning experiences are relevant, inclusive and motivating. Questions on food and drink provide common ground for students, and stimulate engagement and inquiry. Supporting and contextualising this argument is scholarship from higher education studies, and scholarship that defines and describes the characteristics of food studies as a field of enquiry. The article extends a body of scholarship, well represented in TEXT, on the synergetic relation between food and writing, and writing pedagogy.

Handwriting Development in Spanish Children With and Without Learning Disabilities: A Graphonomic Approach

The central purpose of this study was to analyze the dynamics of handwriting movements in real time for Spanish students in early grades with and without learning disabilities. The sample consisted of 120 children from Grades 1 through 3 (primary education), classified into two groups: with learning disabilities and without learning disabilities. The Early Grade Writing Assessment tasks selected for this purpose were writing the alphabet in order from memory, alphabet copying in cursive and manuscript, and allograph selection. The dynamics of these four handwriting tasks were recorded using graphonomic tablets (type Wacom Intuos-4), Intuos Inking pens, and Eye and Pen 2 software. Several events were recorded across four different tasks: velocity, pressure, time invested in pauses, and automaticity. The results demonstrated significant graphonomic variations between groups across grades, depending on the type of task.

Examining Gender Differences in Written Assessment Tasks in Biology: A Case Study of Evolutionary Explanations.

Understanding sources of performance bias in science assessment provides important insights into whether science curricula and/or assessments are valid representations of student abilities. Research investigating assessment bias due to factors such as instrument structure, participant characteristics, and item types are well documented across a variety of disciplines. However, the relationships among these factors are unclear for tasks evaluating understanding through performance on scientific practices, such as explanation. Using item-response theory (Rasch analysis), we evaluated differences in performance by gender on a constructed-response (CR) assessment about natural selection (ACORNS). Three isomorphic item strands of the instrument were administered to a sample of undergraduate biology majors and nonmajors (Group 1: n = 662 [female = 51.6%]; G2: n = 184 [female = 55.9%]; G3: n = 642 [female = 55.1%]). Overall, our results identify relationships between item features and performance by gender; however, the effect is small in the majority of cases, suggesting that males and females tend to incorporate similar concepts into their CR explanations. These results highlight the importance of examining gender effects on performance in written assessment tasks in biology.

Pedagogy for fostering criticality, reflectivity and praxis in a course on teaching for lecturers

Using the concepts of criticality, reflectivity and praxis, the paper presents an analysis of our reflections on participants’ responses to the assessment requirements for a course for lecturers on teaching. The context in which the course is being taught has changed considerably in the last few years in terms of the mode of delivery, as well as the number and diversity of participants. Our analysis has generated insights into ways in which the course is not meeting all the learning needs of the participants, nor preparing them adequately to demonstrate, in writing, their learning. Using insights gained, we suggest pedagogic processes and strategies for ensuring that the course focuses on both writing to learn and learning to write; and for assisting participants to acquire the practices to demonstrate their learning in written assessment tasks, using the requisite literacy including criticality, reflectivity and praxis.

The Use of Mathematics Taxonomy in Developing Statistical Learning Outcomes for Grade 12 Learners

In this paper the use of Mathematics taxonomy in developing statistical learning outcomes (SLO) for Grade 12 learners is investigated. A literature on Mathematics taxonomy and the SLO was reviewed. Further, a case was made for using the Mathematics taxonomy and the SLO in writing assessment tasks in teaching statistics. It is argued that Mathematics taxonomy uses four cognitive levels, knowledge, routine procedures, complex procedures and solving problem to guide all the assessment tasks. Furthermore, it is argued that the use of the three SLO, statistical literacy, reasoning and thinking to distinguish between learning outcomes is helpful in teaching statistics. A qualitative content analysis obtained from the 2010 and 2011 Grade 12 mathematics examination papers was done to examine the use of Mathematics taxonomy in developing the SLO. Results showed that the Mathematics taxonomy should be used in developing statistical literacy, reasoning and thinking. Mathematics teachers should be trained to usethe Mathematics taxonomy and the SLO in statistics assessment tasks.

The Study of Constructive Elements of Writing Assessment Tasks for the Tests of Proficiency in Korean

The Study of Constructive Elements of Writing Assessment Tasks for the Tests of Proficiency in Korean