MyUEVision: an application generating image caption for assisting visually impaired people

Abstract

PurposeVisually impaired people usually struggle with doing daily tasks due to a lack of visual cues. For image captioning assistive applications, most applications require an Internet connection for the image captioning generation function to work properly. In this study, we developed MyUEVision, an application that assists visually impaired people by generating image captions that can work with and without the Internet. This work also involves reviewing some image captioning models for this application.Design/methodology/approachThe author has selected and experimented with three image captioning models for online models and two image captioning models for offline models. The user experience (UX) design was designed based on the problems faced by visually impaired users when using mobile applications. The application is developed for the Android platform, and the offline model is integrated into the application for the image captioning generation function to work offline.FindingsAfter conducting experiments for selecting online and offline models, ExpansionNet V2 is chosen for the online model and VGG16 + long short-term memory (LSTM) is chosen for the offline model. The application is then developed and assessed, and the results show that the application can generate image captions with or without the Internet, providing the best result when having an Internet connection, and the image is captured in good lighting with a few objects.Originality/valueMyUEVision stands out for its both online and offline functionality. This approach ensures the image captioning generator works with or without the Internet, setting it apart as a unique solution to address the needs of visually impaired individuals.