Abstract
Peer review has been used in both online and offline classrooms to inspire creativity, gather feedback, and lessen instructor grading loads, especially for design-based tasks without definitive rubrics. To explore the nuances and quality of peer feedback, we developed UX Factor, a peer grading platform that aims to characterize the behavior of peer reviews and the consistency of the ranking models used to aggregate these reviews. This system harnesses the power of pairwise comparisons to minimize bias and encourage context-driven analysis. We adopted UX Factor in a user interface course of 133 students and teaching assistants (TAs) across 3 different individual design projects over a semester and found that the system was effective in eliciting high-quality feedback. We saw that raters have higher agreement than random preferences, and with at least 15 ratings per submission, a simple average of ratings produced rankings that were consistent to both the raw ratings and other more complex models. These rankings were robust to disagreeable raters and changing class sizes, demonstrating the potential of comparative peer review to match the quality of expert feedback at scale.
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Proceedings of the ACM on Human-Computer Interaction
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
