IMPLEMENTING ASSISTIVE TECHNOLOGY FOR THE VISUALLY IMPAIRED USING DEEP LEARNING, COMPUTER VISION AND NLP

Abstract

Visual impairment can cause difficulties in performing daily activities. Unfamiliarity of places, hindrance in known routes, water puddles, pot holes, stray animals, sudden road incidents etc. decreases the confidence and interaction of the Visually Impaired People (VIP) leading to human assistance. We are presenting a human-like autonomous assistant which is in the form of a stick made with technologies such as Computer Vision, NLP, GPS, Deep Learning and Embedded Systems. VIP can communicate with the stick through Chatbot. He can get answers to general questions like weather status, time of the day, what is my location, where is point A, etc. and specific questions like what is in front of me, in which room I am, provide me the navigation from point A to point B etc.. In situations like an obstacle in the path, pot holes, ascending/descending stairs, approaching people etc., our stick will notify the VIP and will give him the clear path also. The stick will come along with a smart phone based application which will help in keeping a track of the location which can be shared with anyone. We have used CNN to make an object detection model which can classify 20 different objects. A day and night classifier is also included to correctly figure out the time of the day. The stick is equipped with a camera, ultrasonic and wet sensors, GPS module with controller. Real time video through camera will be captured and processed by the controller. Then this will evaluate the feed through object detection model. The outcome can be notified to the VIP through Chatbot. This will enable safety, security, control and more independence to the VIPs.