Acceptability and Effectiveness Analysis of Large Language Model-Based Artificial Intelligence Chatbot Among Arabic Learners

Abstract

This research stems from the broad use of AI based on Large Language Models (LLMs), which many academics find relevant and effective in higher education Arabic language learning. The goal is to confirm these views.This research is a mixed reseach that employs a both of qualitative and quantitative methodologies. The qualitative segment involves observations and literature reviews. Observations involved reviewing how participants used chatbots and carefully checking the accuracy and consistency of platform responses. The quantitative facet utilizes a paired experimental design, encompassing both classical and Bayesian Paired Sample t-Tests analysis. The research encompasses 45 individuals with a proficient understanding of Modern Standard Arabic and no hindrances in comprehending the material. These individuals are enrolled as students at Islamic College (STAI) Al-Anwar Rembang, Indonesia. The results show increased motivation and ease of use with the chatbot in Arabic language learning. However, concerns about the consistency of chatbot content have arisen, affecting participants' confidence in response accuracy of AI. This prompts an evaluation of effectiveness through classical and Bayesian tests, which fail to demonstrate statistically significant variances, even in the adaptive Bayesian probability analysis. These outcomes deviate from previous research on relevance and effectiveness and corroborate preceding studies on academic apprehensions and accuracy enhancements. The researchers advocate for further investigations, especially concerning the accuracy analysis of AI chatbots in Arabic pedagogical contexts.