Threshold optimization of task allocation models in human–machine collaborative scoring of subjective assignments

Abstract

In the exploration of human–machine collaborative scoring of subjective assignment (HMCSSA) problems, it is crucial to note that the workloads assigned to humans and machines are determined by thresholds at various levels of granularity within the human–machine task allocation model. To address this issue, a three-phase framework for optimization and decision making in HMCSSA problems is constructed by combining multi-objective evolutionary algorithms (MOEAs) with multi-attribute decision making method. Specifically, we present a bi-objective threshold optimization model to achieve a trade-off between human costs and scoring fairness in HMCSSA problems. Moreover, four well-known MOEAs are employed for solving the optimization model to search the Pareto optimal thresholds. Meanwhile, the technique for order of preference by similarity to ideal solution (TOPSIS) method is utilized to determine the best human–machine task allocation schemes based on the preferences of decision makers. The numerical experiments are conducted on eight prompts of the ASAP data set to validate the effectiveness and superiority of the proposed framework. In particular, the best task allocation scheme attains the fairness gain of 53.76% with the human participation rate of 21.77% on average.