Human–machine collaborative scoring of subjective assignments based on sequential three-way decisions

Abstract

Automated scoring of subjective assignments (ASSA), one of the most representative applications in the field of artificial intelligence in education, is the process of scoring and assessing essays with the aid of natural language processing technology. Nevertheless, there are a few critical opinions regarding ASSA models. For instance, when essays of high or low quality are fed to an ASSA model, it may result in a great difference between the scores given by the model and human evaluators. To overcome these problems, this study is devoted to the exploration on human–machine collaborative scoring of subjective assignments based on sequential three-way decision. Firstly, we propose a human–machine task allocation model based on sequential three-way decisions (TA-S3WD) to establish a formal framework for human–machine collaborative scoring of subjective assignments. Secondly, we develop a human–machine collaborative subjective assignment scoring model by combining bidirectional long short-term memory (Bi-LSTM) with TA-S3WD model, called HMCS-BLTS. Finally, numerical experiments with intensive comparisons and sensitivity analysis are conducted on the automated student assessment prize (ASAP) data set to show the advantages of the proposed model. Comparative analysis shows that the HMCS-BLTS model achieves higher execution efficiency than seven well-known baseline models in terms of four common evaluation metrics. Moreover, compared with the best-performing baseline model, the HMCS-BLTS model obtains 0.8967 average quadratic weighted kappa and achieves 19.31% improvements on average using just 19.02% of the human workload.