Peer Grades Research Articles

Improving Peer Assessment by Incorporating Grading Behaviors: Models and Practices

Peer assessment, which requires students to evaluate their peers’ submissions, has become the paradigm for solving the grading challenge of large-scale open-ended assignments teachers face in MOOCs. Since peer grades may be biased and unreliable, a group of probabilistic graph models are proposed to improve the estimation of true scores for assignments based on peer grades, by explicitly modeling the bias and reliability of each grader. However, these models assume that graders’ reliability are only impacted by their knowledge/ ability levels while ignoring their grading behaviors. In real life, graders’ grading behaviors (e.g., the time consumed for reviewing an assignment) reflect the seriousness of the graders in the assessment and greatly affect their reliability. Following this intuition, we propose two novel probabilistic graph models, named BPG6 and BPG7, for cardinal peer assessment, which optimize the modeling of the reliability of graders by incorporating various grading behaviors of them. Besides, we develop a peer assessment system, named BPA, which collects abundant grading behaviors of graders and thus facilitates the evaluation of the proposed models and our teaching practices. Experimental results on the collected datasets show the superiority of the proposed models in improving the estimation accuracy of the true scores of assignments by leveraging graders’ grading behaviors. Finally, by analyzing the feedback from teaching practices of peer assessment using the BPA system, we not only prove the usability of the BPA system but also once again demonstrate the effectiveness of the proposed model in peer assessment.

AsPeer: Method to Self and Peer Assessment in Large Online Design Classes

Peer and self-assessment open opportunities to scale assessments in online classrooms. This article reports our experiences of using AsPeer- peer assessment system, with two iterations of a university online class. We observed that peer grades highly correlated with staff assigned grades. It was recorded that, the peer grade of all student submissions within the range of instructor grade averaged to 21.0% and that within the next 2 ranges was 49.0%. We performed three experiments to improve accuracy of peer grading. First, we observed grading bias and introduced a data driven feedback mechanism to inform peers of it. Students aided by feedback were mindful and performed grading with better accuracy. Second, we observed that the rubric lacked efficiency in translating intent to students. Simplified guiding questions improved accuracy in assessment by 89% of students. Third, we encouraged peers to provide personalized qualitative feedback along with rating. We provided them with feedback snippets that addressed common issues. 64% of students responded that the snippets helped them to critically look at submissions before rating.

LearnEval Peer Assessment Platform: Iterative Development Process and Evaluation

Peer assessment is a valuable educational activity, which promotes learner engagement and critical thinking, while also decreasing the time spent by the teachers on evaluating students’ work. Over the past decades, several platforms have been proposed to support the peer assessment process. The current landscape is dominated by special-purpose, context-specific peer evaluation systems, which were designed to support a particular type of learning activity, artifact, or discipline. In this article, we propose an alternative, a generic and comprehensive peer assessment platform, called LearnEval, which is very flexible and able to accommodate a variety of pedagogical scenarios. Distinctive functionalities include: highly configurable peer assessment workflow, several automatic and manual review allocation mechanisms, initial calibration/training options for the students, complex dashboard with statistics, scores and graphical visualizations, as well as an extensive open learner model. The platform was developed in an iterative manner, with several extensions and improvements informed by the findings of the initial pilot studies. The article also reports on a practical application of LearnEval in a project-based learning scenario, with very promising results; multiple perspectives of the peer assessment process are analyzed: student behavior, peer grades validity, and perceived learning experience. Comparative results with an earlier study show increased levels of system usability, grading validity, and student satisfaction.

Fostering college students’ critical thinking skills through peer assessment in the knowledge building community

ABSTRACT Critical thinking is one of the core twenty-first century skills. This study designs the strategy of peer assessment based on the theory of knowledge building to improve a class of 33 undergraduate students’ critical thinking skills in one semester. The context of knowledge building has four phases—idea generation, idea connection, idea improvement, and rising above—which are iterated by three rounds of peer assessment. In this study, the dependent variable was critical thinking skills, and the independent variable was peer assessment. Data sources include critical thinking skills surveyed, peer comments on Knowledge Forum and grades scored by students. The results showed that (1) students’ critical thinking skills were significantly improved; (2) students’ reports have risen in peer assessment; and (3) students’ critical thinking skills are significantly correlated with peer assessment in the knowledge-building community with both peer comments and peer grades. The findings indicated that knowledge building created a positive culture for the promotion of critical thinking skills, comments that expressed strong disagreement could improve students’ critical thinking skills, and the formation of community knowledge played a substantial role in promoting students’ critical thinking skills.

Hybrid and Online Peer Group Grading: Adding Assessment Efficiency While Maintaining Perceived Fairness

ABSTRACT The rate of transition to online instruction in higher education was accelerated by the new reality of the global pandemic, COVID-19. As a result, faculty must adapt by being more efficient to manage more students with less resources and even less personal contact for the foreseeable future. Efficiency can be gained by using group assignments and peer group grading. Peer group grading reduces assessment time although students may be concerned about having their work graded fairly by other students. The peer group grading process and grades are analyzed against the instructor grades over a two-year period. Results demonstrate no significant difference in peer group grades and instructor grades for pre-COVID-19 hybrid and post-COVID-19 online courses. Student survey results also show that students perceive peer group grading as fair in both course formats. Therefore, instructors can feel confident about using peer group grading as a fair and time saving assessment tool.

Leveraging Cognitive Diagnosis to Improve Peer Assessment in MOOCs

A major challenge faced by popular massive open online courses (MOOCs) is the assessment of large-scale open-ended assignments submitted by students. Recently, peer assessment has become a mainstream paradigm that helps grade open-ended assignments on a large scale. In peer assessment, students also become graders in grading a small number of their peers' assignments, and the peer grades are then aggregated to predict a true score for each assignment. The collected peer grades are usually inaccurate because graders have different reliabilities and biases. To improve accuracy, several probabilistic graph models have been proposed to model the reliability and bias of each grader. However, none of these models consider graders' competency information in the assignments to be graded, which has been found to be very effective. We propose two new probabilistic graph models to improve the accuracy of cardinal peer assessments based on the well-accepted cognitive diagnosis technique. Specifically, the cognitive diagnosis model DINA is applied to determine grader competency based on historical tests or assignments. Then, this information is used to optimize the modeling of grader reliability in each of the proposed models. Moreover, an effective model inference algorithm is proposed to infer true scores of assignments. Experimental results based on real world datasets show that the two proposed models outperform state-of-the-art models and that consideration of grader competency contributes to improved score estimation.

Using Continuous Peer Evaluation in Team-Based Engineering Capstone Projects: A Case Study

Scholarship of Application. Background: Capstone courses constitute the culminating experience in engineering curricula. A core characteristic is the student team and project-based nature, with many team deliverables. The ensuing difficulty is to fairly assess the individual team member's contribution toward the team effort. Contribution: The method presented here is based on weekly peer evaluation of the individual team member's contribution, which subsequently yields a participation factor (PF), in turn scaling the team grade in accordance with the individual student performance. Intended Outcome: This method allows the timely adjustment of the individual student effort to match the group expectations and rewards high-performing students by making higher individual grades than the team average possible, while penalizing underperforming free riders by not benefiting from other student's performance. Application Design: The method allows students to continuously calibrate their own and their teammate's expectations and improve their peer score by adjusting their individual efforts. This feedback also requires students to practice professional communication, and in particular giving and receiving critical feedback, and thus is highly aligned with industry needs. Findings: Results evaluated over three iterations of the assessment process indicate a weak positive correlation (0.26) of the peer evaluation with the individual instructor-graded deliverables as well as the individual student grade point average (GPA, 0.23). Further, survey-based data indicate student agreement that the PF is a fair reflection of the individual performance, and a neutral perceived overall assessment system effectiveness, with reported primary barriers being the difficulty in assigning fair peer grades and of open, critical discussion.

SSPA: an effective semi-supervised peer assessment method for large scale MOOCs

ABSTRACT Peer assessment has become a primary solution to the challenge of evaluating a large number of students in Massive Open Online Courses (MOOCs). In peer assessment, all students need to evaluate a subset of other students’ assignments, and then these peer grades are aggregated to predict a final score for each student. Unfortunately, due to the lack of grading experience or the heterogeneous grading abilities, students may introduce unintentional deviations in the evaluation. This paper proposes and implements a semi-supervised peer assessment method (SSPA) that incorporates a small number of teacher’s gradings as ground truth, and uses them to externally calibrate the procedure of aggregating peer grades. Specifically, each student’s grading ability is directly (if students have common peer assessments with teacher) or indirectly (if students have no common peer assessments with teacher) measured with the grading similarity between the student and teacher. Then, SSPA utilizes the weighted aggregation of peer grades to infer the final score of each student. Based on both real dataset and synthetic datasets, the experimental results illustrate that SSPA performs better than the existing methods.

Improving essay peer grading accuracy in massive open online courses using personalized weights from student's engagement and performance

AbstractMost massive open online courses (MOOC) use simple schemes for aggregating peer grades, taking the mean or the median, or compute weights from information other than the instructor's opinion about the students' knowledge. To reduce the difference between the instructor and students' aggregated scores, some proposals compute specific weights to aggregate the peer grades. In this work, we analyse the use of students' engagement and performance measures to compute personalized weights and study the validity of the aggregated scores produced by these common functions, mean, and median, together with two others from the information retrieval field, the geometric and harmonic means. To test this procedure, we have analysed data from a MOOC about Philosophy. The course had 1,059 students registered, and 91 participated in a peer review process that consisted in writing an essay and rating three of their peers using a rubric. We compared the aggregation scores obtained using weighted and nonweighted versions of the functions. Our results show that the correlation between the aggregated scores and the instructor's grades can be improved in relation to peer grading, when using the median and weights are computed according to students' performance in chapter tests.

"New Media in Education" MOOC: Improving Peer Assessments of Students’ Plans and their Innovativeness

MOOC courses in the fields of new media, entrepreneurship and innovation require pedagogical developers to create a clear and measureable peer assessment process of the innovative plans and projects developed by students. The objectives of our action research are to improve evaluations of students’ plans and their innovativeness in a MOOC based on project-based learning (PBL). We analyzed 789 written peer assessments and grades of 89 PBL plans, which were submitted as part of the requirements in a new MOOC course 'New Media in Education' targeting student teachers. Correlations between final peer project grades and other assessment categories were calculated. A regression analysis model indicates that innovativeness and compatibility of student plans to educational needs are the strongest predictors final peer grades. Based on the research findings, we propose potential improvements in the peer assessment process, including additional guidelines for the assessment of innovativeness and other categories of PBL plans based on new media and digital pedagogy.

Peer Victimization, Self-Concept, and Grades: The Mediating Role of Self-Concept

ABSTRACTThis study examined the relations between specific forms of peer victimization, math self-concept, and math grades of 740 students. Results showed that, in boys, all forms of victimization did not affect grades, while in girls, a highly perceived social exclusion impacted grades. In boys, a highly perceived humiliation negatively affected self-concept, whereas a highly perceived social exclusion positively affected self-concept. Boys’ self-concept mediated the relations between social exclusion and grades as well as between humiliation and grades. In girls, a highly perceived happy slapping positively affected self-concept, and self-concept mediated the relation between a highly perceived happy slapping and grades.

Improving Peer Grading Reliability with Graph Mining Techniques

Peer grading is an approach increasingly adopted for assessing students in massive on-line courses, especially for complex assignments where automatic assessment is impossible and the ability of tutors to evaluate and provide feedback at scale is limited. Unfortunately, as students may have different expertise, peer grading often does not deliver accurate results compared to human tutors. In this paper, we describe and compare different methods, based on graph mining techniques, aimed at mitigating this issue by combining peer grades on the basis of the detected expertise of the assessor students. The possibility to improve these results through optimized techniques for assessors’ assignment is also discussed. Experimental results with both synthetic and real data are presented and show better performance of our methods in comparison to other existing approaches.

Assessing the “I” in group work assessment: State of the art and recommendations for practice

Introduction: The use of group work assessment in medical education is becoming increasingly important to assess the competency of collaborator. However, debate continues on whether this does justice to individual development and assessment. This paper focuses on assessing the individual component within group work.Method: An integrative literature review was conducted and complemented with a survey among representatives of all medical schools in the Netherlands to investigate current practices.Results: The 14 studies included in our review show that an individual component is mainly assessed by peer assessment of individual contributions. Process and product of group work were seldom used separately as criteria. The individual grade is most often based on a group grade and an algorithm to incorporate peer grades. The survey provides an overview of best practices and recommendations for implementing group work assessment.Discussion: The main pitfall when using peer assessment for group work assessment lies in differentiating between the group work process and the resulting product of the group work. Hence, clear criteria are needed to avoid measuring only effort. Decisions about how to weigh assessment of the product and peer assessment of individual contribution should be carefully made and based on predetermined learning goals.

Evaluating Maps in a Massive Open Online Course

New forms of cartographic education are becoming possible with the synthesis of easy to use web GIS tools and learning platforms that support online education at a massive scale. The internet classroom can now support tens of thousands of learners at a time, and while some common types of assessments scale very easily, others face significant hurdles. A particular concern for the cartographic educator is the extent to which original map designs can be evaluated in a massive open online course (MOOC). Based on our experiences in teaching one of the first MOOCs on cartography, we explore the ways in which very large collections of original map designs can be assessed. Our methods include analysis of peer grades and qualitative feedback, visual techniques to explore design methods, and quantitative comparison between expert ratings and peer grades. The results of our work suggest key challenges for teaching cartography at scale where instructors cannot provide individual feedback for every student.

Assessment of a new evaluation tool in a doctoral candidate seminar course

ObjectiveTo determine if a more objective and descriptive evaluation tool more closely aligned faculty and peer grades than a previous rubric in a seminar course. Additional objectives were to evaluate grade changes between the tools, determine the association of student characteristics on grades, and to assess faculty and student perceptions of the effectiveness of the new tool. MethodsFaculty and peer grades were compared to each other in 2013 and 2014. Total combined grades for 2013 were compared to those of 2014. Students and presentation evaluators who participated in the 2014 course were surveyed for demographics and perceptions of the tool. Statistical analyses were performed for grade comparisons, demographic data, and perception data. ResultsFaculty and peer grades were not more closely aligned using the new tool (p < 0.001). Total combined grades for 2013 and 2014 were not more closely aligned with the new tool (p = 0.222). From the demographic data collected, only female gender and a desire to pursue postgraduate education or training had a significant effect on grades (p < 0.001 and p = 0.003, respectively). Overall perception responses for faculty and students were positive as they relate to rubric helpfulness. ConclusionsThe new evaluation tool was positively perceived to be more objective among faculty and students; however, it did not result in more closely aligned faculty and peer grades. Lastly, the data from this study indicate that few known modifiable predictors of student success in a seminar course exist currently.

The assessment cycle: a model for learning through peer assessment

This paper advances a model describing how peer assessment supports self-assessment. Although prior research demonstrates that peer assessment promotes self-assessment, the connection between these two activities is underspecified. This model, the assessment cycle, draws from theories of self-assessment to elaborate how learning takes place through peer assessment. The model is applied to three activity structures described in the literature to analyse their potential to support learning by promoting self-assessment. Broadly speaking, the model can be used to understand learning that takes place in a variety of peer assessment activities: marking/grading, analysis, feedback, conferencing and revision. This approach contrasts most studies on peer assessment, which have focused on calibration of instructor and peer grades, rather than learning opportunities.

Probabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs

With the enormous scale of massive open online courses (MOOCs), peer grading is vital for addressing the assessment challenge for open-ended assignments or exams while at the same time providing students with an effective learning experience through involvement in the grading process. Most existing MOOC platforms use simple schemes for aggregating peer grades, e.g., taking the median or mean. To enhance these schemes, some recent research attempts have developed machine learning methods under either the cardinal setting (for absolute judgment) or the ordinal setting (for relative judgment). In this paper, we seek to study both cardinal and ordinal aspects of peer grading within a common framework. First, we propose novel extensions to some existing probabilistic graphical models for cardi- nal peer grading. Not only do these extensions give su- perior performance in cardinal evaluation, but they also outperform conventional ordinal models in ordinal eval- uation. Next, we combine cardinal and ordinal models by augmenting ordinal models with cardinal predictions as prior. Such combination can achieve further performance boosts in both cardinal and ordinal evaluations, suggesting a new research direction to pursue for peer grading on MOOCs. Extensive experiments have been conducted using real peer grading data from a course called “Science, Technology, and Society in China I” offered by HKUST on the Coursera platform.

A Workshop Series Using Peer-grading to Build Drug Information, Writing, Critical-thinking, and Constructive Feedback Skills

To utilize a skills-based workshop series to develop pharmacy students' drug information, writing, critical-thinking, and evaluation skills during the final didactic year of training. A workshop series was implemented to focus on written (researched) responses to drug information questions. These workshops used blinded peer-grading to facilitate timely feedback and strengthen assessment skills. Each workshop was aligned to the didactic coursework content to complement and extend learning, while bridging and advancing research, writing, and critical thinking skills. Attainment of knowledge and skills was assessed by rubric-facilitated peer grades, faculty member grading, peer critique, and faculty member-guided discussion of drug information responses. Annual instructor and course evaluations consistently revealed favorable student feedback regarding workshop value. A drug information workshop series using peer-grading as the primary assessment tool was successfully implemented and was well received by pharmacy students.

Peer and self assessment in massive online classes

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Peer, professor and self-evaluation of class participation

The purpose of this project was to determine the validity of peer and self-evaluations of class participation compared to professors' class participation grades. Students (N = 96) evaluated themselves and their classmates on class participation on a four-point scale and students were required to assign grades in a normalized distribution. Relative to faculty evaluations, the bias and precision of the peer grades were 0.48 points and 36.3 per cent (p &lt; 0.05) and self-evaluations scores were -0.48 and 77.5 per cent (p &lt; 0.05).There was no correlation between a student's grade point average and his/her opinion of this process (R = 0.02). Students did not like peer assessment using forced distribution of grades.