Online Peer Assessment Research Articles

Online peer assessment in Galenic Pharmacy: enhancing evaluative judgement in higher education

ABSTRACT Assessment influences how students define their priorities and their commitment to the learning process. Assessment strategies can empower students to actively engage in metacognitive processes, fostering cross-curricular competencies. Mastery of these competencies not only enhances deep and meaningful learning but also prepares learners for the challenges of the ever-evolving knowledge field. However, developing evaluative judgement, the ability to critically and autonomously judge the quality of one’s own work and that of others, is essential but challenging. The purpose of this study was to design and assess an online educational experience for Galenic Pharmacy students (N = 339) during the 2021–2022 academic year of the Degree of Pharmacy. Beyond content acquisition, the primary goal was to foster evaluative judgement as a pivotal component of the ‘learning to learn’ competence. A complex task with iterative deliverables was proposed, using peer assessment as the central tool for the development of evaluative judgement. Students were required to give presentations on methods of administering medicines and had iterative deliverables. They underwent multiple rounds of peer feedback. The task as well as peer assessment process were compulsory for all the students. The participating students voluntarily answered an ad hoc online questionnaire in relation to their perception of the overall experience of peer assessment. The outcomes showcased the positive impacts of peer assessment based on the roles of assessor and assessed. An improvement in feedback quality was observed from one iteration to another, and an enhancement of critical judgement was evident. Enhancing assessment literacy might be essential for both educators and students. For educators, this would allow them to set criteria more aligned with competencies, whereas students might place higher value on these practices and actively engage with the learning process. Such engagement is crucial for promoting lifelong autonomous and self-regulated learning.

Investigating behavioral patterns to facilitate performance predictions during online peer assessment through learning analytics approach

Insights into student behavioral patterns yield benefits for both educators and learners. Nevertheless, only a few studies have investigated the behavioral patterns of high-scoring (HS) and low-scoring (LS) students to facilitate predictions during peer assessment (PA). Therefore, we performed learning analytics to explore the behavioral patterns of HS and LS students in 52 university students from affective, cognitive, and metacognitive perspectives as these students engage in online PAs. The results indicated that on the affective dimension, HS students tended to exhibit negative affection while LS students tended to display positive affection. On the cognitive dimension, HS students demonstrated more intricate transformations compared to the LS students. Regarding the metacognitive dimension, LS students more likely reflected upon and accepted the reviews provided by others than the HS. Additionally, the findings also revealed that HS students were supported by focused social networks and overall stable activity engagement, whereas the LS students were less engaged at the beginning of the activity but had evolving social networks and were more engaged in the later stages of the activity. These findings offer valuable insights for educators in effectively predicting student performance based on behavioral patterns while providing new perspectives to support instructional decision-making processes.

Online peer assessment in paragraph writing its implementation and students’ constraints

Peer Assessment (PA) has recently been gaining increasing popularity in higher education due to its advantages in engaging students, and its use in English as a Foreign Language (EFL) writing classes is very common in Indonesia. The implementation of PA became more necessary when the government regulated emergency remote teaching in response to the COVID-19 crisis. This study examines the implementation of an online peer assessment (OPA) and the constraints encountered by the students in the OPA of a paragraph writing class. To achieve the objectives of this study, an embedded mixed method design was employed, which consists of an interview, observations, documentation, and a survey. The respondents in this study are a teacher of the writing class and 30 English Education Department students of a state university. The gleaned qualitative data were analyzed thematically while the quantitative data were analyzed by finding the percentages generated by Google Forms. The results show that the teacher made use of the three stages of the OPA: preparation, instruction, and evaluation. The students faced three constraints: the quality of the feedback, impartiality, and the clarity of the assessment criteria. It is recommended that when performing an OPA, teachers provide clear instructions regarding the stages and pay attention to the issues experienced by students.

Effects of Descriptive Peer Assessment on the Learning of Middle and High School Students in South Korea

This study examined the impact of online peer assessment based on descriptive questions on student learning in Korea. The validity of the scores for student-written descriptive responses rated by middle and high school students were compared to teacher grading. The impact of peer assessment activities based on descriptive questions on student learning was also investigated, as was investigated students’ perceptions of peer assessment. First-year middle and high school students studying science were recruited. The results reveal a high level of agreement between peer assessment scores of middle and high school students and teacher grades. The peer assessment activity based on descriptive questions enhanced students’ academic achievement. Students stated peer assessment activity was effective for conceptual learning. This study confirmed the effectiveness of peer assessment activities based on descriptive questions in students’ conceptual learning.

A systematic review of educational online peer-review and assessment systems: charting the landscape

Over the past two decades, there has been an explosion of innovation in software tools that encapsulate and expand the capabilities of the widely used student peer assessment. While the affordances and pedagogical impacts of traditional in-person, “paper-and-pencil” peer assessment have been studied extensively and are relatively well understood, computerized (online) peer assessment introduced not only shifts in scalability and efficiency, but also entirely new capabilities and forms of social learning interactions, instructor leverage, and distributed cognition, that still need to be researched and systematized. Despite the ample research on traditional peer assessment and evidence of its efficacy, common vocabulary and shared understanding of online peer-assessment system design, including the variety of methods, techniques, and implementations, is still missing. We present key findings of a comprehensive survey based on a systematic research framework for examining and generalizing affordances and constraints of online peer-assessment systems. This framework (a) provides a foundation of a design-science metatheory of online peer assessment, (b) helps structure the discussion of user needs and design options, and (c) informs educators and system design practitioners. We identified two major themes in existing and potential research—orientation towards scaffolded learning vs. exploratory learning and system maturity. We also outlined an agenda for future studies.

Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment

AbstractBackgroundPeer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning. However, with the emergence of numerical grades and textual feedback generated by peers, it is necessary to detect the reliability of the large amount of peer assessment data, and then develop an effective automated grading model to analyse the data and predict learners' learning results.ObjectivesThe present study aimed to propose an automated grading model with reliability detection.MethodsA total of 109,327 instances of peer assessment from a large‐scale teacher online learning program were tested in the experiments. The reliability detection approach included three steps: recurrent convolutional neural networks (RCNN) was used to detect grade consistency, bidirectional encoder representations from transformers (BERT) was used to detect text originality, and long short‐term memory (LSTM) was used to detect grade‐text consistency. Furthermore, the automated grading was designed with the BERT‐RCNN model.Results and ConclusionsThe effectiveness of the automated grading model with reliability detection was shown. For reliability detection, RCNN performed best in detecting grade consistency with an accuracy rate of 0.889, BERT performed best in detecting text originality with an improvement of 4.47% compared to the benchmark model, and LSTM performed best with an accuracy rate of 0.883. Moreover, the automated grading model with reliability detection achieved good performance, with an accuracy rate of 0.89. Compared to the absence of reliability detection, it increased by 12.1%.ImplicationsThe results strongly suggest that the automated grading model with reliability detection for large‐scale peer assessment is effective, with the following implications: (1) The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results. (2) This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. (3) This solution could shift educational institutions from labour‐intensive grading procedures to a more efficient educational assessment pattern, allowing for more investment in supporting instructors and learners to improve the quality of peer feedback.

Sustainable education on improving the quality of peer assessment: design and implementation of an online deep learning-based peer assessment system

PurposeAs global issues such as climate change, economic growth, social equality and the wealth gap are widely discussed, education for sustainable development (ESD) allows every human being to acquire the knowledge, skills, attitudes and values necessary to shape a sustainable future. It also requires participatory teaching and learning methods that motivate and empower learners to change their behavior and take action for sustainable development. Teachers have begun rating pupils based on peer assessment for open evaluation. Peer assessment enables students to transition from passive to active feedback recipients. The assessors improve critical thinking and encourage introspection, resulting in more significant recommendations. However, the quality of peer assessment is variable, resulting in reviewers not recognizing the remarks of other reviewers, therefore the benefits of peer assessment cannot be fulfilled. In the past, researchers frequently employed post-event questionnaires to examine the effects of peer assessment on learning effectiveness, which did not accurately reflect the quality of peer assessment in real time.Design/methodology/approachThis study employs a multi-label model and develops a self-feedback system in order to use the AIOLPA system in the classroom to enhance students' learning efficacy and the validity of peer assessment.FindingsThe research findings indicate that the better peer assessment through the rapid feedback system, for the evaluator, encourages more self-reflection and attempts to provide more ideas, so bringing the peer rating closer to the instructor rating and assisting the evaluator. Improve self-evaluation and critical thinking for the evaluator, peers make suggestions and comments to help improve the work and support the growth of students' learning effectiveness, which can lead to more suggestions and an increase in the work’s quality.Originality/value ESD consequently promotes competencies like critical thinking, imagining future scenarios and making decisions in a collaborative way. This study builds an online peer assessment system with a self-feedback mechanism capable of classifying peer comments, comparing them with scores in a consistent manner and providing prompt feedback to critics.

How to promote students to share more knowledge with peers?

Education experts are trying to encourage students to contribute knowledge among them through online peer assessment so as to enhance the understanding of knowledge. As an online knowledge community, peer assessment will also fail due to the reluctance of students to return the favor and share knowledge. Many studies on this topic have highlighted the importance of contribution of knowledge for students which forms a virtuous feedback loop for the impression of knowledge. However, it is unclear how what influences its development. Motivated by this, this study explored why students contribute knowledge to others through online peer assessment. Empirical panel count data were collected from Peer review platform. The results from a negative binomial regression model with user fixed effects indicate that student’s peer recognition and organization interaction have a positive impact on his knowledge-contribution behaviors. The findings can help guide the development and operation of peer assessment.

Student rating consistency in online peer assessment from the perspectives of individual and class

Most of previous studies examined the consistency among two or three student raters giving scores to different peers (i.e. ratees), whereas few of them researched the consistency among more student raters giving scores to different peers (i.e. ratees). The present study aimed to see whether there is a consistency among student raters giving scores to peers in an online assessment activity. Participants of the study were 266 graduate students, with 137 males and 129 females, from 27 classes in a department at some university. Each student was required to give a presentation graded by the peers anonymously. From to the perspective of class, the result of the Kendall coefficient of concordance showed that a proportion of more than half-classes presented a significant correlation with a moderate or strong consistency among student raters giving scores to peers, and thus the number of raters or peers should be more for a higher reliability. The result of Chi-square test from the perspective of individual revealed that students held a consistent standard when giving scores to peers' presentations. Finally the implications for theories and practices are proposed.

Peer Assessment of Oral Presentations in EFL Classes: Assessors’ View

The study focuses on peer assessment (PA) used in teaching English as a foreign language. It aimed to identify students’ attitudes towards PA in oral presentations, using online PA forms and an online post PA questionnaire. Thematic analysis of the data show positive students’ attitudes towards PA, as it helped them to understand the task requirements better, be more focused, and improve their own presentation skills. The drawbacks were linked to stress during the assessment.

What predicts variation in reliability and validity of online peer assessment? A large‐scale cross‐context study

AbstractBackgroundFor peer assessment, reliability (i.e., consistency in ratings across peers) and validity (i.e., consistency of peer ratings with instructors or experts) are frequently examined in the research literature to address a central concern of instructors and students. Although the average levels are generally promising, both reliability and validity can vary substantially from context to context. Meta‐analyses have identified a few moderators that are related to peer assessment reliability/validity, but they have lacked statistical power to systematically investigate many moderators or disentangle correlated moderators.ObjectivesThe current study fills this gap by addressing what variables influence peer assessment reliability/validity using a large‐scale, cross‐context dataset from a shared online peer assessment platform.MethodsUsing multi‐level structural equation models, we examined three categories of variables: (1) variables related to the context of peer assessment; (2) variables related to the peer assessment task itself; and (3) variables related to rating rubrics of peer assessment.Results and ConclusionsWe found that the extent to which assessment documents varied in quality on the given rubric played a central role in mediating the effect from different predictors to peer assessment reliability/validity. Other variables that are significantly associated with reliability and validity included: Education Level, Language, Discipline, Average Ability of Peer Raters, Draft Number, Assignment Number, Class Size, Average Number of Raters, and Length of Rubric Description. The results provide information to guide practitioners on how to improve reliability and validity of peer assessments.

Offline and online peer assessment in computer engineering: Insights from a 5-year experience

Offline and online peer assessment in computer engineering: Insights from a 5-year experience

Online peer-assessment quality control: A proactive measure, validation study, and sensitivity analysis

Despite the generally positive learning effects of peer-assessment (PA), there are some undesirable signs and troublesome behavior needed to be addressed (e.g., biased judgments, superficial-level engagement). Anchoring on the concept of individual accountability and assessment validity, while attending to other important elements (including both subjective and objective criteria, individuals’ differentiation ability across the points on the assessment scale, and completion rate), a measure was proposed for PA quality control. This measure consists of four indexes/variables and enables the quantification and appraisal of the peer-assessor’s performance. A validation study involving two classes of undergraduate students (N = 103) engaged in the assessment of peers’ oral presentations was conducted to evaluate the effects of the proposed measure. The statistically significant results confirmed the efficacy of the proposed measure to promote better online PA performance. Additionally, to assess its applicability, sensitivity analysis targeting different aspects of our proposed measure (i.e., different plausible weighting for each of the associated indexes/variables, different correlation values between peers’ and the teacher’s ratings, and different bound values for the scale) was conducted. All significant results of the 281 test cases further substantiated the applicability and efficacy of our proposed measure under various assessment schemes for PA quality control.

Procedures for online peer assessment: Assessing algorithm problems in school mathematics for future teachers

This action research study explored how peer assessment can help students and teachers evaluate algorithm problem-solving skills in mathematics. The study used a self- and peer-assessment activity in Moodle to assess 18 out of 40 Grade X students (10 boys and eight girls) from a Kathmandu Valley, Nepal school. The students solved algorithm problems in arithmetic, algebra, statistics, and geometry. The study followed the action research methodology of planning, implementing, assessing, and discussing the interventions and outcomes. The results show that workshop activity can engage students in solving algorithm problems in mathematics. The study also discusses how cognitive and constructivist theories can explain some of this activity's unique aspects and potential uses. Moreover, the study highlights the benefits and challenges of self- and peer-assessment in mathematics for enhancing students' interest and abilities in the classroom. The study suggests students can develop analytical and evaluative skills using evaluation criteria to assess their peers' work. The study also implies that students are proactive, critical, and collaborative learners who can use self-and peer assessment to improve their mathematical abilities to solve algorithm problems in the classroom.

“Free selection and invitation” online peer assessment of undergraduates’ research competencies, flow, motivation and interaction in a research methods course

“Free selection and invitation” online peer assessment of undergraduates’ research competencies, flow, motivation and interaction in a research methods course

Assessment of the validity of peer scores and peer feedback in an online peer assessment platform (Kritik)

Background and purposeOnline peer assessment platforms aim to improve the accuracy of peer scores (numerical scores based on a rubric) and hold students accountable for peer feedback (written comments). We evaluated the validity of peer scores and peer feedback using the online platform, Kritik. Educational activity and settingThe course was a two-credit hour online infectious diseases pharmacotherapy elective consisting of twelve third-year students enrolled in a four-year doctor of pharmacy program. Students analyzed patient cases and created video presentations of their therapeutic care plan weekly. Each student scored three peers' presentations using a rubric and provided peer feedback in Kritik. The instructor independently scored the presentations. The students' presentation scores, which was the weighted average of three peers' scores, was compared to the instructor's score. Students also rated the peer feedback they received using two Likert-type scales (feedback-on-feedback [FoF] ratings). Two faculty rated 97 randomly selected peer feedback (written comments) and recorded their FoF ratings separately. Students completed an anonymous course evaluation and exit survey. FindingsThe Pearson correlation coefficient between weighted peer scores and instructor scores for 91 presentations was r = 0.880. There was fair agreement in FoF ratings between students and faculty based on weighted kappa. All students would recommend the course and had a positive experience with peer assessment and the platform. SummaryWeighted peer scores strongly correlated with instructor scores, and students held each other accountable for peer feedback in Kritik. Our findings should be confirmed in different contexts and settings.

Peer assessment of professionalism attributes

Peer assessment can support the development of professionalism by providing feedback that enables learners to reflect on their professional behavioural attributes. We developed and implemented an innovative online peer assessment and feedback tool. Students were encouraged to nominate 12 peer assessors to anonymously conduct their assessment. Assessors were presented with a list of 32 adjectives that described professional behavioural attributes within four domains (integrity, conscientiousness, agreeableness and resilience) and asked to rate the student by selecting a minimum of two adjectives in each domain and to provide free-text comments. The feedback was presented as a collated word cloud and free-text comments. All students had the opportunity to discuss their profiles with a staff member. Our mixed-methods evaluation found that all students participated, and they valued the peer assessment and feedback process. Although the assessment was formative and confidential, students were reluctant to provide negative comments about their peers. 'Disengaged', 'aloof' and 'argumentative' were the most likely negative adjectives that indicated students with low-level professionalism concerns. Future development will focus on introducing students who can act as peer champions for the process and repeating the peer assessment over time to identify the change in professionalism development.

Online Peer Assessment for Learning: Findings from Higher Education Students

Assessment practices in the higher education (HE) context have undergone profound changes over recent years, particularly regarding their purpose, strategies, and available resources. This exploratory study seeks to analyze, through the perceptions of HE students, the contribution and adequacy of an assessment for learning strategy, namely, online peer assessment (OPA), inspired by the conceptual framework of the PrACT Model, a framework which aims to contribute to the dissemination of alternative assessment practices. The main data collection technique used was the survey questionnaire and the study participants (n = 16) were students from a higher education institution in Portugal. Results point to the lack of student experience in the practice of OPA and are discussed in conformity with the dimensions of the PrACT framework. OPA is considered, from the student’s perspective, an adequate alternative digital assessment strategy, contributing to student motivation as well as to the development of cognitive, metacognitive, and digital skills.

Effects of online peer assessment on higher‐order thinking: A meta‐analysis

AbstractOnline peer assessment (OPA) has been increasingly adopted to develop students' higher‐order thinking (HOT). However, there has not been a synthesis of research findings on its effects. To fill this gap, 17 papers (published from 2000 to 2022) that reported either a comparison between a group using OPA (n = 7; k = 22) and a control group or a pre–post comparison (n = 10; k = 17) were reviewed in this meta‐analysis. The overall effect of OPA on HOT was significant (g = 0.76). Furthermore, OPA exerted more significant effects on convergent HOT (eg, critical thinking, reasoning and reflective thinking; g = 0.97) than on divergent HOT (eg, creativity and problem‐solving; g = 0.38). Reciprocal roles and anonymity were found to positively moderate the impacts of OPA on HOT, although their moderating effects were not statistically significant because of small sample size of studies in the analysis. The results of the meta‐analysis reinforce the arguments for regarding OPA as a powerful learning tool to facilitate students' HOT development and reveal important factors that should be considered when adopting OPA to enhance students' HOT. Practitioner notesWhat is already known about this topic Online peer assessment (OPA) has significant positive impacts on learning achievement. OPA has been regarded as a potential approach to cultivating students' higher‐order thinking (HOT) but has not been proved by meta‐analysis. OPA should be carefully designed to maximise its effectiveness on learning. What this paper adds OPA has been proved to significantly positively influence students' HOT via meta‐analysis. OPA exerted more significant effects on convergent HOT than on divergent HOT. The potential of reciprocal roles and anonymity for moderating the impacts of OPA on HOT should not be underestimated. Implications for practice and/or policy OPA could be a wise choice for practitioners when they help students to achieve a balanced development of HOT dispositions and skills. Students' divergent HOT can be encouraged in their uptake of peer feedback and by allowing them autonomy in deciding assessment criteria. OPA with design elements of reciprocal roles and anonymity has great potential to promote students' HOT.

A Bibliometric Analysis of Peer Assessment in Online Language Courses

As a popular strategy in collaborative learning, peer assessment has attracted keen interest in academic studies on online language learning contexts. The growing body of studies and findings necessitates the analysis of current publication trends and citation networks, given that studies in technology-enhanced language learning are increasingly active. Through a bibliometric analysis involving visualization and citation network analyses, this study finds that peer assessment in online language courses has received much attention since the COVID-19 outbreak. It remains a popular research topic with a preference for studies on online writing courses, and demonstrates international and interdisciplinary research trends. Recent studies have led peer assessment in online language courses to more specific research topics, such as critical factors for improving students’ engagement and feedback quality, unique advantages in providing online peer assessment, and designs to enhance peer assessment quality. This study also provides critical aspects about how to effectively integrate educational technologies into peer assessment in online language courses. The findings in this study will encourage future studies on peer assessment in online learning, language teaching methods, and the application of educational technologies.