Improving Arabic Text to Image Mapping Using a Robust Machine Learning Technique

Abstract

In this paper, we introduce an approach to automatically convert simple modern standard Arabic children’s stories to the best representative images that can efficiently illustrate the meaning of words. It is a kind of imitating the imaginative process when children read a story, yet a great challenge for a machine to achieve it. For simplification issues, we apply several techniques to find the images and we associate them with related words dynamically. First, we apply natural language processing techniques to analyze the text in stories and we extract keywords of all characters and events in each sentence. Second, we apply an image captioning process through a pre-trained deep learning model for all retrieved images from our multimedia database as well as the Google search engine. Third, using sentence similarities, most significant images are retrieved back by selecting top- $k$ highest similarity values. It is worth mentioning that using the captioning process, to rank top- $k$ images, has shown reasonable precision values as per our preliminary results. The option to refine or validate the ranked images to compose the final visualization for each story is also provided to ensure a flexible and safe learning environment.