Three Approaches for Detecting Direct Output Cheating in Program Online Judge Systems

Abstract

Program online judge (POJ) systems allow students to view questions, submit solution code, and receive scores automatically via the web. Most POJs use test cases for scoring. When a POJ is scored by test case pass rate or a problem that has only one test case, students can usually score by providing the direct output of the test cases (direct output cheating). Currently, there is only one work on detecting such cheating. However, its precision is very low. To solve this problem, three novel approaches are proposed to detect direct output cheating: (i) Line Statistics, which computes the proportion of output calls against other statements; (ii) the control flow graph (CFG) Search computes the maximum similarity between the CFG of a program and that of known samples; (iii) abstract syntax tree (AST) Search identifies cheating by matching rules that are summarized from ASTs of previously detected cheating attempts. A student’s code is marked as cheating if the similarity exceeds a predefined threshold; and a program is detected as cheating if the proportion exceeds a predefined threshold. The proposed approaches and three well-known code plagiarism detection tools (JPlag, Sherlock, and SIM) were evaluated using 100,000 submissions for 1153 problems from a POJ based on the C programming language. The F1 scores of these approaches were determined as 0.9752 (AST Search), 0.9440 (CFG Search), 0.7405 (Line Statistics), 0.6446 (JPlag), 0.1587 (Sherlock), and 0.0076 (SIM), respectively. The result indicates that (i) AST Search is most suitable for the detection of direct output cheating; (ii) traditional code search or plagiarism detection methods based on similarity calculations are not effective for complex cheat detection because these cheats are highly similar to normal code.