Illustrate It! An Arabic Multimedia Text-to-Picture m-Learning System

Abstract

Multimedia learning is the process of building mental representation from words associated with images. Due to the intuitiveness and vividness of visual illustration, many texts to picture systems have been proposed. However, we observe some common limitations in the existing systems, such as the retrieved pictures may not be suitable for educational purposes. Also, finding pedagogic illustrations still requires manual work, which is difficult and time-consuming. The commonly used systems based on the best keyword selection and the best sentence selection may suffer from loss of information. In this paper, we present an Arabic multimedia text-to-picture mobile learning system that is based on conceptual graph matching. Using a knowledge base, a conceptual graph is built from the text accompanied with the pictures in the multimedia repository as well as for the text entered by the user. Based on the matching scores of both conceptual graphs, matched pictures are assigned relative rankings. The proposed system demonstrated its effectiveness in the domain of Arabic stories, however, it can be easily shifted to any educational domain to yield pedagogical illustrations for organizational or institutional needs. Comparisons with the current state-of-the-art systems, based on the best keyword selection and the best sentence selection techniques, have demonstrated significant improvements in the performance. In addition, to facilitate educational needs, conceptual graph visualization and visual illustrative assessment modules are also developed. The conceptual graph visualization enables learners to discover relationships between words, and the visual illustrative assessment allows the system to automatically assess the performance of a learner. The profound user studies demonstrated the efficiency of the proposed multimedia learning system.