Peer Assessment Data Research Articles

The impacts of peer assessment on critical thinking competence: An epistemic network analysis

Critical thinking competence has recently become a popular topic in education research; however, it is insufficiently researched in tourism education. This study seeks to evaluate the effectiveness of peer assessment in developing critical thinking skills in a Chinese context. Peer assessment data were collected through group news presentations and individual assignments of college students studying tourism management. An epistemic network analysis was conducted to examine the role of peer assessment in students' critical thinking competence. The findings show that students’ critical thinking skills significantly improve as peer assessment activities are conducted, with higher-level critical thinking skills clearly emerging at a later stage of peer assessment activities.

Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment

AbstractBackgroundPeer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning. However, with the emergence of numerical grades and textual feedback generated by peers, it is necessary to detect the reliability of the large amount of peer assessment data, and then develop an effective automated grading model to analyse the data and predict learners' learning results.ObjectivesThe present study aimed to propose an automated grading model with reliability detection.MethodsA total of 109,327 instances of peer assessment from a large‐scale teacher online learning program were tested in the experiments. The reliability detection approach included three steps: recurrent convolutional neural networks (RCNN) was used to detect grade consistency, bidirectional encoder representations from transformers (BERT) was used to detect text originality, and long short‐term memory (LSTM) was used to detect grade‐text consistency. Furthermore, the automated grading was designed with the BERT‐RCNN model.Results and ConclusionsThe effectiveness of the automated grading model with reliability detection was shown. For reliability detection, RCNN performed best in detecting grade consistency with an accuracy rate of 0.889, BERT performed best in detecting text originality with an improvement of 4.47% compared to the benchmark model, and LSTM performed best with an accuracy rate of 0.883. Moreover, the automated grading model with reliability detection achieved good performance, with an accuracy rate of 0.89. Compared to the absence of reliability detection, it increased by 12.1%.ImplicationsThe results strongly suggest that the automated grading model with reliability detection for large‐scale peer assessment is effective, with the following implications: (1) The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results. (2) This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. (3) This solution could shift educational institutions from labour‐intensive grading procedures to a more efficient educational assessment pattern, allowing for more investment in supporting instructors and learners to improve the quality of peer feedback.

An Item Response Theory Approach to Enhance Peer Assessment Effectiveness in Massive Open Online Courses

Massive open on-line courses (MOOCs) are effective and flexible resources to educate, train, and empower populations. Peer assessment (PA) provides a powerful pedagogical strategy to support educational activities and foster learners' success, also where a huge number of learners is involved. Item response theory (IRT) can model students' features, such as the skill to accomplish a task, and the capability to mark tasks. In this paper the authors investigate the applicability of IRT models to PA, in the learning environments of MOOCs. The main goal is to evaluate the relationships between some students' IRT parameters (ability, strictness) and some PA parameters (number of graders per task, and rating scale). The authors use a data-set simulating a large class (1,000 peers), built by a Gaussian distribution of the students' skill, to accomplish a task. The IRT analysis of the PA data allow to say that the best estimate for peers' ability is when 15 raters per task are used, with a [1,10] rating scale.

The good, the loyal or the active? MPs’ parliamentary performance and the achievement of static and progressive career ambitions in parliament

AbstractAlthough often theorized, empirical research on the relationship between MPs’ parliamentary behaviour and their chances to realize career ambitions is relatively scarce. This article holistically analyses the effect of MPs’ (1) party loyalty, (2) activity level and (3) the quality of their parliamentary work on MPs’ prospects for re-election and their promotion to higher parliamentary office. Based on a unique combination of behavioural and peer assessment data on 325 federal and regional MPs in Belgium (2014–2019), we find that particularly MPs’ loyalty and activity level improve their career prospects in the subsequent term, in contrast to more qualitative aspects of their parliamentary work. These findings provide important new insights into how and to what degree legislators are rewarded for their parliamentary performance.

Using learning analytics to understand student perceptions of peer feedback

Peer assessment (PA) is the process of students grading and giving feedback to each other's work. Learning analytics is a field focused on analysing educational data to understand and improve learning processes. Using learning analytics on PA data has the potential to gain new insights into the feedback giving/receiving process. This exploratory study focuses on backward evaluation, an under researched aspect of peer assessment, where students react to the feedback that they received on their work. Two aspects are analysed: 1) backward evaluation characteristics depending on student perception of feedback that they receive on their work, and 2) the relationship between rubric characteristics and backward evaluation. A big dataset (N = 7,660 records) from an online platform called Peergrade was analysed using both statistical methods and Epistemic Network Analysis. Students who found feedback useful tended to be more accepting by acknowledging their errors, intending to revise their text, and praising its usefulness, while students who found the feedback less useful tended to be more defensive by expressing that they were confused about its meaning, critical towards its form and focus, and in disagreement with the claims. Moreover, students mostly suggested feedback improvement in terms of feedback specificity, justification and constructivity, rather than kindness. The paper concludes by discussing the potential and limitations of using LA methods to analyse big PA datasets.

Extraction, Processing and Visualization of Peer Assessment Data in Moodle

Situated in the intersection of two emerging trends, online self- and peer assessment modes and learning analytics, this study explores the current landscape of software applications to support peer assessment activities and their necessary requirements to complete the learning analytics cycle upon the information collected from those applications. More particularly, the study focuses on the specific case of Moodle Workshops, and proposes the design and implementation of an application, the Moodle Workshop Data EXtractor (MWDEX) to overcome the data analysis and visualization shortcomings of the Moodle Workshop module. This research paper details the architecture design, configuration, and use of the application, and proposes an initial validation of the tool based on the current peer assessment practices of a group of learning analytics experts. The results of the small-scale survey suggest that the use of software tools to support peer assessment is not so extended as it would initially seem, but also highlight the potential of MWDEX to take full advantage of Moodle Workshops.

Predicting Substance Use and Deviant Behavior from Prosociality and Sociability in Adolescents.

Social competence is commonly considered an important factor that impedes maladaptive development because individuals who lack adequate competence to direct or control their behaviors in social situations are likely to display problems. Despite the belief that social competence may be a multi-dimensional construct, existing research has not explored the unique contributions of its different aspects to development. The present two-wave longitudinal study examined relations of prosociality and sociability, two major aspects of social competence, with problem behaviors in a sample of adolescents. Peer assessment data on prosociality and sociability and self-report data on substance use and deviant behavior were obtained from a sample of adolescents in middle (N = 657, mean age = 13 years) and high (N = 675, mean age = 16 years) schools in China. The results showed that prosociality negatively predicted later problem behaviors, whereas sociability positively predicted later problem behaviors. In addition, gender and grade moderated the associations between sociability and problem behaviors. More specifically, the positive association between sociability and later substance use was significant for boys, but not for girls. Sociability was also more strongly associated with later substance use in middle school than in high school. The results indicate that prosociality and sociability may have different functional meanings in adolescent development and have implications for practical work with adolescents who display problem behaviors.

The PILAR Model as a Measure of Peer Ratings of Collaboration Viability in Small Groups

The PILAR (prospects, involved, liked, agency, respect) model provides a dynamical systems perspective on collaboration. Two studies are performed using peer assessment data, both testing empirical support for the five Pillars that constitute members’ perceptions of collaboration viability (CoVi). The first study analyses peer assessment data collected online from 458 first-year engineering students (404 males; 54 females). A nine-item instrument was inherited from past year’s usage in the course, expanded with four additional items to elaborate upon the agency and liked Pillars. Exploratory factor analysis was conducted on student responses to test whether they thematically aligned to constructs consistent with the five Pillars. As anticipated, twelve of the thirteen items grouped into five components, each aligned with a Pillar, providing empirical evidence that the five Pillars represent perceptions of collaboration. The second study replicated the first study using a retrospective analysis of 87 items included in the Comprehensive Assessment of Team Member Effectiveness (CATME) peer assessment tool. The associated factor analyses resulted in five components and conceptual alignment of these components with Pillars was evident for three of five CATME components. We recommend a peer assessment instrument based upon PILAR as potentially more parsimonious and reliable than an extensive list of behaviours, such as employed by CATME. We also recommend including items that target inter-rater bias, which is aligned with the liked Pillar, that instruments such as CATME exclude.

Variability in students' evaluating processes in peer assessment with calibrated peer review

AbstractThis study investigated students' evaluating process and their perceptions of peer assessment when they engaged in peer assessment using Calibrated Peer Review. Calibrated Peer Review is a web‐based application that facilitates peer assessment of writing. One hundred and thirty‐two students in an introductory environmental science course participated in the study. Two self‐reported surveys and a focus group interview were administered during the semester. The peer assessment data and demographic information were collected at the end of the semester. Although the study results support the agreement between peers and an expert, the variations in a group and individual level were found, in particular, when students evaluated mid‐quality or low‐quality writings regardless of their reviewing ability. Students tended to perceive that the process of evaluating peers' and own writings was helpful in their learning. Further, students' positive perceptions of peer assessment were associated with their understanding of the values of peer assessment tasks and their perceptions of achieving the course goal. We concluded that instructors should provide specific guidelines for how to decide a rating, use actual students' essays instead of instructor‐developed samples to train students and require written explanation for rubric questions to reduce variation in students' ratings and promote learning.

Supporting Mediated Peer-Evaluation to Grade Answers to Open-Ended Questions

We show an approach to semi-automatic grading of answers given by students to open ended questions (open answers). We use both peer-evaluation and teacher evaluation. A learner is modeled by her Knowledge and her assessments quality (Judgment). The data generated by the peer- and teacher- evaluations, and by the learner models is represented by a Bayesian Network, in which the grades of the answers, and the elements of the learner models, are variables, with values in a probability distribution. The initial state of the network is determined by the peer-assessment data. Then, each teacher’s grading of an answer triggers evidence propagation in the network. The framework is implemented in a web-based system. We present also an experimental activity, set to verify the effectiveness of the approach, in terms of correctness of system grading, amount of required teacher's work, and correlation of system outputs with teacher’s grades and student’s final exam grade.

Item Response Theory for Peer Assessment

As an assessment method based on a constructivist approach, peer assessment has become popular in recent years. However, in peer assessment, a problem remains that reliability depends on the rater characteristics. For this reason, some item response models that incorporate rater parameters have been proposed. Those models are expected to improve the reliability if the model parameters can be estimated accurately. However, when applying them to actual peer assessment, the parameter estimation accuracy would be reduced for the following reasons. 1) The number of rater parameters increases with two or more times the number of raters because the models include higher-dimensional rater parameters. 2) The accuracy of parameter estimation from sparse peer assessment data depends strongly on hand-tuning parameters, called hyperparameters. To solve these problems, this article presents a proposal of a new item response model for peer assessment that incorporates rater parameters to maintain as few rater parameters as possible. Furthermore, this article presents a proposal of a parameter estimation method using a hierarchical Bayes model for the proposed model that can learn the hyperparameters from data. Finally, this article describes the effectiveness of the proposed method using results obtained from a simulation and actual data experiments.

Infusing assessment into mathematics content courses for pre-service elementary school teachers

The present study aims to explore the use of assessment in mathematics content courses for future elementary school teachers. Analysis of self assessment data on mathematical understanding and peer assessment data on oral mathematical presentation showed that pre-service teachers had a balanced understanding of procedural knowledge and problem solving. Conceptual understanding was not in the structure of pre-service teachers’ mathematical knowledge. Understandings of conceptual knowledge, procedural knowledge, and problem solving had no meaningful effects on gains in mathematics performance. Aspects of oral mathematical presentation were associated with improved understanding of procedural knowledge and in particular conceptual knowledge. The result of the study calls for a conceptual approach to mathematical knowledge and sufficient mathematical problem solving in college-level mathematics content courses and in particular the infusion of assessment into college-level mathematics education for pre-service teachers.

Relationship Between Peer Assessment During Medical School, Dean’s Letter Rankings, and Ratings by Internship Directors

BackgroundIt is not known to what extent the dean’s letter (medical student performance evaluation [MSPE]) reflects peer-assessed work habits (WH) skills and/or interpersonal attributes (IA) of students.ObjectiveTo compare peer ratings of WH and IA of second- and third-year medical students with later MSPE rankings and ratings by internship program directors.Design and ParticipantsParticipants were 281 medical students from the classes of 2004, 2005, and 2006 at a private medical school in the northeastern United States, who had participated in peer assessment exercises in the second and third years of medical school. For students from the class of 2004, we also compared peer assessment data against later evaluations obtained from internship program directors.ResultsPeer-assessed WH were predictive of later MSPE groups in both the second (F = 44.90, P < .001) and third years (F = 29.54, P < .001) of medical school. Interpersonal attributes were not related to MSPE rankings in either year. MSPE rankings for a majority of students were predictable from peer-assessed WH scores. Internship directors’ ratings were significantly related to second- and third-year peer-assessed WH scores (r = .32 [P = .15] and r = .43 [P = .004]), respectively, but not to peer-assessed IA.ConclusionsPeer assessment of WH, as early as the second year of medical school, can predict later MSPE rankings and internship performance. Although peer-assessed IA can be measured reliably, they are unrelated to either outcome.

Peer assessments of performance and ability: An exploratory meta-analysis of statistical artifacts and contextual moderators

The validity of peer assessments as an additional source of information in personnel decision-making was assessed via Hunter and Schmidt's (1989) meta-analytic approach. Fifty-six correlations on peer assessments of ability and performance were located. After correcting for sampling error and imperfect reliability, the mean correlation was estimated to be .69, with a 95% confidence interval of .46 to .92. Also, contextual factors hypothesized to affect observed validities were evaluated. Although much potentially important information is not reported in available studies, the data suggested that use of peer assessment data, time of acquaintance, and type of criterion affect observed validity. Implications for future research are discussed.