CBET: design and evaluation of a domain-specific chatbot for mobile learning

Abstract

The popularity of mobile devices and conversational agents in recent years has seen wide use of chatbots in different educational scenarios. In relation to the advances in mobile devices and conversational agents, there are few research works concerning the design and evaluation of domain-specific chatbots to fulfill the demand of mobile learning. To address this issue, we propose an agent-based conceptual architecture to develop a domain-specific chatbot for mobile learning. We extend the open-domain DeepQA agent to make it sensitive to restricted domain questions by building a domain-specific gate, and employ WeChat as user interface. To evaluate our chatbot, subjective and objective criteria are employed to assess its effectiveness. Additionally, its usability evaluation proceeds with system usability scale questionnaire and net promoter score simultaneously. In total, 18 domain experts participated in the evaluation of effectiveness, and 52 participants were involved in the evaluation of usability. Based on the evaluation results, we conclude that our chatbot can serve as an effective information retrieval tool in a specific domain. The perceived usability of our chatbot tends to be moderate and marginal and has positively affected the promotion of our chatbot for mobile learning. This paper contributes to the educative application of chatbots in specific subject fields.