Reliability Of Peer Assessment Research Articles

Reliability and Validity of Peer Assessment at Tutorless PBL Classes

Introduction: Learning behaviors in PBL and team-based learning (TBL) at Mie University Faculty of Medicine have been measured with peer assessments in addition to tutor assessments. The change in class components posed a question on reliability of peer assessments at tutorless sessions in TBL. Methods: Correlations between peer and tutor assessments in the PBL program comprised of tutor and tutorless sessions were analyzed with data from 377 students. The three domains for student’s performance, “self-learning in preparation”, “active participation in group discussion” and “cooperative attitude at group work”, were collected at 4 UNITs with the formal PBL and at 4 UNITs with the mixture of PBL and TBL. Results: No statistically significant difference between peer and tutor assessments was found in all three domains in units with the mixture of PBL and TBL. The correlation analysis between peer and tutor assessments in units with the mixture of PBL and TBL demonstrated strong positive correlations in self-learning in preparation (r=0.62) and in active participation in group discussion (r= 0.63), and a weak positive correlation in cooperative attitude at group works (r=0.35). Discussion: This study revealed that students evaluated their group-peers at the units comprised of both tutor and tutorless sessions similarly to those comprised of only tutor sessions. Peer assessments were well correlated with tutors’ assessment at the units comprised of tutor and tutorless session. Our study suggests that TBL might be expanded in the situation where teaching resource is limited.

Exploring the reliability and its influencing factors of peer assessment in massive open online courses

AbstractAs an innovative evaluation tool, peer assessment is essential in Massive Open Online Courses (MOOCs). In both formative and summative peer assessments in MOOCs, providing reliable feedback is crucial in enhancing learning outcomes. Peer assessment has been highlighted as a reliable tool in both traditional classrooms and small‐scale online learning contexts by a wealth of studies. Factors impacting the result of peer assessment have also been identified. However, such exploration in MOOCs is very limited. Through an examination of 5,722 assignment submissions and 56,794 review scores collected from 18 peer assessments in three courses on Chinese University MOOC platform, this study investigated the inter‐reliability of peer assessment in MOOCs utilizing Krippendorff's alpha and ICC[1]. It also revealed factors affecting peer assessment reliability in MOOCs. The results of this study show that students tended to mark extreme scores, and peer assessment in MOOCs seemed unreliable. In addition, the reliability of the peer assessment in these MOOCs was inversely correlated to the number of reviewers per assignment and reviews completed per reviewer. The assignment type also matters; peer assessment using e‐portfolios format was found more reliable than those using papers and proposals. Suggestions on how to improve the reliability of peer assessments in MOOCs are also provided. Practitioner notesWhat is already known about this topic Both the formative and summative peer assessment mode, providing reliable feedback is needed, which is of great value for improving the quality of online learning. The reliability and influence factors of peer assessment has been highlighted by a wealth of studies in traditional classrooms and small‐scale online learning contexts. There are still limited explorations in MOOCs. What the paper adds The students tended to give extreme scores in peer assessment in MOOC, and the peer assessment in MOOC may was not particularly reliable. The Assignment type being rated affected the reliability, and peer assessment using e‐portfolios format was found more reliable than those using papers and proposals. The reliability of peer assessment in MOOC was negative related to the number of reviewers per assignment and the reviews completed per reviewer. Implications for practice and/or policy Avoid using peer assessment as a summative assessment method, and should be assigned relatively low weights to final grades. Using papers and proposals as learning assignments can appropriately reduce the proportion of peer assessment in the final grade. Training on how to identify medium quality work should be provided. Instructional strategies should be designed to encourage learners to actively participate in peer assessments and thoroughly assess.

Effects of the Assessment Criteria Levels on Academic Achievement, Interest, and Reliability of Peer Assessment in Peer Assessments

Effects of the Assessment Criteria Levels on Academic Achievement, Interest, and Reliability of Peer Assessment in Peer Assessments

Authentic-Peer Assessment Instrument To Measure The Ability Of A Chemistry Teacher Candidate Evaluation On Basic Chemistry Small Skill Laboratory Works

The dominance of paper and pencil assessment tests in chemistry small skill laboratory works make assessments in the affective and psychomotor less attention. This research aims to develop an authentic-peer assessment instrument to measure the ability of a prospective evaluation of chemistry teachers on basic chemistry small skill laboratory works. The method used research and development. The subjects were students of the second semester of chemistry education 2018 at Walisongo State Islamic University Semarang. Data collection instrument in the form of sheets validation instrument to measure the validity and reliability of peer assessment and questionnaire responses sheet learners. The results of authentic-peer assessment developed very validly in the category with a value of 31. The internal consistency reliability Spearman-Brown gained 0.78 means indicates that the authentic assessment instruments developed have a strong level of consistency. The measurement results evaluation capabilities candidate chemicals are known educator excellent ability group with a percentage of 31.45%, a good ability with the percentage of 47.48%, enough ability with the percentage of 15.79%, and less ability with the percentage of 5.28%. The questionnaire results obtained by 82 percentage points. The results of that study concluded that authentic assessment developed meet the category of valid, reliable, and practical.

PEER ASSESSMENT AT TERTIARY LEVEL

Autonomous second language learning (SLL) is a topic of focus in every educational setting and almost universally supported by institutions and scholars in the whole world. One aspect of autonomous learning is peer evaluation (Harmer, 2001; Benson, 2001; Cottrall, 1999; Everhard&Murphy) Seeking to make students more autonomous leaners and take more responsibility for their learning, as well as assess the effect of peer assessment, on language acquisition, peer assessment has been introduced to student of the final semester of their studies. This paper reports initial results from a study into the effectiveness of peer assessment in an EFL (English as a Foreign Language) classroom instruction, in a methodology course, conducted in the second semester of the academic year 2017/2018, at the Department of English Language and Literature, at the University of Tetovo, Macedonia. Applying frequency statistics, Fraenkel&Wallen, (2003), the study has analyzed peer assessment; in contrast to teacher assessment of student presentations. Content analysis (Leedy & Ormord, 2005) to identify themes biases and meaning, the study investigated the validity and reliability of peer assessment. Convenience sampling among students enrolled in the seventh semester identified a non-random sample of 20 students representing 41% of the possible cohort. The findings of the study revealed that students were not prepared to assess each other and rather acted on their personal feelings towards their peers. In conclusion, it is argued that research designs involving an extended period of time and with a larger sample will be needed to confirm these results.

PEER ASSESSMENT AT TERTIARY LEVEL

Autonomous second language learning (SLL) is a topic of focus in every educational setting and almost universally supported by institutions and scholars in the whole world. One aspect of autonomous learning is peer evaluation (Harmer, 2001; Benson, 2001; Cottrall, 1999; Everhard&Murphy) Seeking to make students more autonomous leaners and take more responsibility for their learning, as well as assess the effect of peer assessment, on language acquisition, peer assessment has been introduced to student of the final semester of their studies. This paper reports initial results from a study into the effectiveness of peer assessment in an EFL (English as a Foreign Language) classroom instruction, in a methodology course, conducted in the second semester of the academic year 2017/2018, at the Department of English Language and Literature, at the University of Tetovo, Macedonia. Applying frequency statistics, Fraenkel&Wallen, (2003), the study has analyzed peer assessment; in contrast to teacher assessment of student presentations. Content analysis (Leedy & Ormord, 2005) to identify themes biases and meaning, the study investigated the validity and reliability of peer assessment. Convenience sampling among students enrolled in the seventh semester identified a non-random sample of 20 students representing 41% of the possible cohort. The findings of the study revealed that students were not prepared to assess each other and rather acted on their personal feelings towards their peers. In conclusion, it is argued that research designs involving an extended period of time and with a larger sample will be needed to confirm these results.

The effects of awareness training on the reliability of peer assessment, team activity satisfaction, and perception on peer assessment in team-based learning

The effects of awareness training on the reliability of peer assessment, team activity satisfaction, and perception on peer assessment in team-based learning

Where angels fear to tread: online peer-assessment in a large first-year class

In the context of widening participation, large classes and increased diversity, assessment of student learning is becoming increasingly problematic in that providing formative feedback aimed at developing student writing proves to be particularly laborious. Although the potential value of peer assessment has been well documented in the literature, the associated administrative burden, also in relation to managing anonymity and intellectual ownership, makes this option less attractive, particularly in large classes. A potential solution involves the use of information and communication technologies to automate the logistics associated with peer assessment in a time-efficient way. However, uptake of such systems in the higher education community is limited, and research in this area is only beginning. This case study reports on the use of the Moodle Workshop module for formative peer assessment of students’ individual work in a first-year introductory macro-economics class of over 800 students. Data were collected through an end-of-course evaluation survey of students. The study found that using the feature-rich Workshop module not only addressed many of the practical challenges associated with paper-based peer assessments, but also provided a range of additional options for enhancing validity and reliability of peer assessments that would not be possible with paper-based systems.

Using peer assessment of fieldwork to enhance students’ practical training

Fieldwork training is a key component of several practical disciplines. In this study, students’ peer assessment of fieldwork is explored as a method to improve their practical training. Peer assessment theories are first discussed. A framework for peer assessment of fieldwork is proposed, and the steps taken for preparation of students for this task are discussed. A developed marking, feedback and moderation tool of assessment are presented. Application of peer assessment in the field was investigated over a period of two years in one undergraduate unit in the geospatial discipline as an example. Reliability of peer assessment was estimated by measuring the difference between assessments carried out by groups of peer assessors, and its validity was measured by comparing students’ marks with those given by tutors. Results show that students have gained from the peer assessment process, mainly as a formative form of assessment, by better understanding and endeavouring to achieve the objectives of field tasks. Tutors use differences among assessments made by groups of students compared to tutors’ assessments to identify field components that need better explanation of their content and assessment criteria.

Effects of peer-assessed feedback, goal setting and a group contingency on performance and learning by 10–12-year-old academy soccer players

Background: Coaches developing young talent in team sports must maximise practice and learning of essential game skills and accurately and continuously assess the performance and potential of each player. Relative age effects highlight an erroneous process of initial and on-going player assessment, based largely on subjective opinions of game performance. A more objective assessment of each player's progress by measuring essential skills during practice is warranted to guide and improve coaching and assessment of talented children. Purpose: To measure the effects of the coaching intervention on performance and learning during a group technical practice. A second purpose was to repeatedly measure each player's passing, first touch and an awareness response and to measure the reliability of peer assessment as an immediate source of objective feedback. Setting: Weekly coaching sessions at an English professional soccer club academy. A total of 22 practices occurred during the second half of a soccer season. Participants: Academy boys aged 10–12 years, with five players participating throughout the study. The coach was a European ‘A’ licence holder and an experienced physical education teacher. Intervention: The coach and players defined the practice requirements and how to perform each skill successfully. Players were taught to observe, assess and record the performance of their peers. After the practice, players charted their objective scores and bonus game play was awarded based on self-set goal attainment. Research design: A single-subject, multiple-baseline experiment was used to assess the effects of the intervention on performance of the three skills and the reliability of the performance measures. The design and intervention enabled the repeated measurement of all players’ performance and learning during each phase and condition of the experiment. Data collection and analysis: Baseline data were collected from video recording of practice and intervention data were collected from recording and live by a player and the coach. Percent correct scores for each skill were plotted on graphs for visual analysis. Agreements between live and primary video data were calculated to evaluate the reliability of peer-assessed scores. Findings: The intervention was shown to improve the level and consistency of performance in all five participants, which, for most players and skills, was maintained when the intervention was removed. Players were willing and able to reliably assess their peers and collect objective data on each player's performance. They enjoyed receiving and charting their scores and reported a greater confidence in these skills in competitive situations. Conclusions: The findings suggest this formal accountability, feedback and reinforcement procedure provides a promising method to maximise practice and learning and accurately assess the progress of talented youth soccer players. Further research is recommended to examine the congruent validity of performance measures from more complex learning tasks to discriminate talent and tackle relative age effects. Implications for future talent development coaching and coach education include the potential for repeatable and measurable systems (a technology) to track performance and learning during coaching–practice to help develop and determine the most able children.

Using Computer-Assisted Synchronous Peer Assessment Program in the EFL Formal Presentation Class

In order to examine viability of implementing peer assessment practice in an EFL classroom setting, the study addressed two research questions related to (1) reliability of peer assessment and (2) development of peer assessing behavior across two assessments. Fifty-six undergraduate students at a university in Korea participated in the study by both performing individual presentations and assessing their peer performances. The findings on the first assessment showed that peer assessment was different from instructor assessment―the students rated much higher than the instructor. In the second assessment, however, the student evaluators rated peer presentation quality quite similarly to the way the instructor did. Looking at the development of student ability to assess peers over certain period of time, the results indicate that student rating behavior of the first assessment significantly differs from that of the second assessment; the students became more critical in evaluating peer presentations based on better understanding of assessment criteria in the second assessment. It is claimed that student ability to assess peers can be developed.

Peer Review of the Quality of Care

Peer assessments have traditionally been used to judge the quality of care, but a major drawback has been poor interrater reliability. To compare the interrater reliability for outcome and process assessments in a population of frail older adults and to identify systematic sources of variability that contribute to poor reliability. Eight sites participating in a managed care program that integrates acute and long-term care for frail older adults. A total of 313 frail older adults. Retrospective review of the medical record with 180 charts randomly assigned to 2 geriatricians, 2 geriatric nurse practitioners, or 1 geriatrician and 1 geriatric nurse practitioner and 133 charts randomly assigned to either a geriatrician or a geriatric nurse practitioner. Interrater reliabilities for structured implicit judgments about process and outcomes for overall care and care for each of 8 tracer conditions (eg, arthritis). Outcome measures had higher interrater reliability than process measures. Five outcome measures achieved fair to good reliability (more than 0.40), while none of the process measures achieved reliabilities more than 0.40. Three factors contributed to poorer reliabilities for process measures: (1) an inability of reviewers to differentiate among cases with respect to the quality of management, (2) systematic bias from individual reviewers, and (3) systematic bias related to the professional training of the reviewer (ie, physician or nurse practitioner). Peer assessments can play an important role in characterizing the quality of care for complex patients with multiple interrelated chronic conditions, but reliability can be poor. Strategies to achieve adequate reliability for these assessments should be applied. These strategies include emphasizing outcomes measurement, providing more structured assessments to identify true differences in patient management, adjusting systematic bias resulting from the individual reviewer and their professional background, and averaging scores from multiple reviewers. Future research on the reliability of peer assessments should focus on improving the ability of process measures to differentiate among cases with respect to the quality of management and on identifying additional sources of systematic bias for both process and outcome measures. Explicit recognition of factors influencing reliability will strengthen efforts to develop sound measures for quality assurance.