A Consideration of the Use of Plagiarism Tools for Automated Student Assessment

Abstract

In this paper, the authors evaluate the flexibility and richness of two well-established text analysis plagiarism tools, through a consideration of the use of plagiarism detection software as a mechanism for the automated assessment of student-created narrative in a virtual learning environment (VLE). The authors are currently engaged in a project creating a prototype VLE, using technologies for multilevel and multiplayer games, based on the inherent support such an environment would provide for constructivist learning, engagement, and contextual socialization. Progress between levels in the VLE will be based on the creation, by the student, of a narrative linking together a number of conceptual elements obtained through game-play at that level. Support for the narrative creation process will help the student to contextualize the conceptual elements, providing the necessary linking elements or themes to enable the student to produce a coherent description of their understanding of the concepts. A particular challenge in such environments is the need for fast, real-time feedback to students to maintain the level of engagement and to support the game-play metaphor. Additionally, the student must be able to make as many attempts to progress as they need and it will be their decision when and how often to submit for assessment. Since the student narrative will be in a textual form and can therefore be related to a sample solution narrative, generated by the author of the level within the learning environment, the idea of using plagiarism detection software as the mechanism for automated comparison and assessment was considered appropriate for investigation. While the limitation of such tools would appear to be that they are seeking direct copies of text elements, the authors wanted to investigate whether they offered sufficient richness and fuzziness to detect common conceptually-linked texts. The initial decision was to experiment with text-analytic tools, since they are both widely used and readily available. The tools chosen were TurnItIn, a commercial tool provided to the U.K. higher education community by the U.K. Joint Information Systems Committee (JISC), and VALT/VAST, a set of tools created at the Centre for Interactive Systems Engineering at London South Bank University, London, U.K., the workings of which are based on recognized and well-published research. An experiment using a small group of students in a traditional assessment situation was carried out, and is described in detail. The rationale for this approach was that there is not yet a fully working prototype of the VLE in which to carry out such an experiment, but that the conditions necessary to test the hypothesis that plagiarism tools could be utilized for such a purpose could be replicated sufficiently to make such an experiment viable. The results of the experiment demonstrated neither a correlation between the sample solution and student solutions, nor any correlation between the individual student solutions, proving the null hypothesis. This result demonstrates that these tools are not useful for the development of automated assessment within the VLE, and the authors are now giving consideration to the use of lexical analysis/tokenizer and other tools. However, it also suggests that these text-analytic plagiarism tools are too firmly focused on direct copy, which does raise the question of whether or not they offer enough richness and fuzziness to detect a sophisticated plagiarism attempt using, for example, text replacement tools. An ongoing close relationship between research in automated assessment and plagiarism detection is also proposed, to achieve mutual benefit.