AI Based Virtual Mouse with Hand Gesture and AI Voice Assistant Using Computer Vision and Neural Networks

Abstract

Abstract: The use of hand gesture recognition in controlling virtual devices has become popular due to the advancement of artificial intelligence technology. A hand gesture-controlled virtual mouse system that utilizes AI algorithms to recognize hand gestures and translate them into mouse movements is proposed in this paper. The system is designed to provide an alternative interface for people who have difficulty using a traditional mouse or keyboard. The proposed system uses a camera to capture images of the user’s hand, which are processed by an AI algorithm to recognize the gestures being made. The system is trained using a dataset of hand gestures to recognize different gestures. Once the gesture is recognized, it is translated into a corresponding mouse movement, which is then executed on the virtual screen. The system is designed to be scalable and adaptable to different types of environments and devices. All the input operations can be virtually controlled by using dynamic/static hand gestures along with a voice assistant. In our work we make use of ML and Computer Vision algorithms to recognize hand gestures and voice commands, which works without any additional hardware requirements. The model is implemented using CNN and mediapipe framework. This system has potential applications like enabling hand-free operation of devices in hazardous environments and providing an alternative interface for hardware mouse. Overall, the hand gesturecontrolled virtual mouse system offers a promising approach to enhance user experience and improve accessibility through human-computer interaction.