Traditional Input Devices Research Articles

Prototype of a Virtual Assistant System Integrated with AI

This research focuses on the development of a prototype virtual assistant system integrated with artificial intelligence (AI) using the Rational Unified Process (RUP) method. The system is designed to improve user flexibility and efficiency by allowing interaction through voice commands, removing the need for traditional input devices. Python was selected as the primary programming language due to its robust capabilities in handling AI-driven applications. The system utilizes Whisper API for speech recognition, enabling the virtual assistant to accurately interpret voice inputs. Additionally, the integration of Chat GPT API allows the assistant to process and generate responses in a natural, context-aware manner. The combination of these technologies is expected to enhance user experience by making the system more intuitive and seamless, applicable to both daily tasks and complex business environments. The RUP method, structured into phases such as inception, elaboration, construction, and transition, was applied to ensure that the development process was iterative, flexible, and aligned with user needs. The results indicate that the integration of Whisper API with Chat GPT API significantly improves the quality and accuracy of voice-based interaction, streamlining system operation while minimizing the need for complex graphical interfaces. This research demonstrates the potential of voice-driven AI systems in increasing overall operational efficiency.

Gesture speak: Hands-Free Computer Control with Hand Gestures and Voice Commands

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.

Iris Unleashed: Revolutionizing Mouse Control through OpenCV

Abstract: Background: In today's modern world, personal computers have become an essential part of our daily routines, serving various purposes like work, education, and entertainment. However, for individuals with limited mobility, using traditional input devices such as keyboards and mouse can be quite challenging. In such cases, alternative input methods that utilize different abilities, such as eye movements, are preferred. Objectives: The proposed system begins by capturing an image with a camera and using OpenCV code to detect the pupil's position in the eye. This information serves as a reference point for the user to control the cursor by moving their eyes left and right.

Virtual Painting through Hand Gestures: A Machine Learning Approach

The hand gesture paint application represents an innovative venture in the realm of computer vision, harnessing the capabilities of the MediaPipe library and OpenCV to develop a sophisticated painting tool driven by hand gestures. By leveraging these powerful frameworks, the application enables users to express their creativity through intuitive gestures captured by a webcam. At its core, the work seeks to redefine the conventional painting experience by providing users with a seamless interface controlled entirely by hand movements. Through the integration of the MediaPipe library and OpenCV, users can effortlessly draw, erase, and switch between various drawing tools, all with the fluidity of natural hand gestures. By eliminating the need for traditional input devices, the hand gesture paint application offers a novel approach to digital art creation, empowering users to engage directly with their canvas in a more intuitive and immersive manner. Through this innovative blend of technology and creativity, the application opens up new avenues for artistic expression, inviting users to explore the boundless possibilities of digital painting through the simple movement of their hands.

Controlling Media Player with Hands: A Transformer Approach and a Quality of Experience Assessment

In this article, we propose a Hand Gesture Recognition (HGR) system based on a novel deep transformer (DT) neural network for media player control. The extracted hand skeleton features are processed by separate transformers for each finger in isolation to better identify the finger characteristics to drive the following classification. The achieved HGR accuracy (0.853) outperforms state-of-the-art HGR approaches when tested on the popular NVIDIA dataset. Moreover, we conducted a subjective assessment involving 30 people to evaluate the Quality of Experience (QoE) provided by the proposed DT-HGR for controlling a media player application compared with two traditional input devices, i.e., mouse and keyboard. The assessment participants were asked to evaluate objective (accuracy) and subjective (physical fatigue, usability, pragmatic quality, and hedonic quality) measurements. We found that (i) the accuracy of DT-HGR is very high (91.67%), only slightly lower than that of traditional alternative interaction modalities; and that (ii) the perceived quality for DT-HGR in terms of satisfaction, comfort, and interactivity is very high, with an average Mean Opinion Score (MOS) value as high as 4.4, whereas the alternative approaches did not reach 3.8, which encourages a more pervasive adoption of the natural gesture interaction.

Vision Based Gesture Recognition

The use of computers has become an integral part of our daily lives, and they are utilized in various fields. However, traditional input devices such as a mouse and keyboard can sometimes be limiting. Hand gestures can provide a more natural and efficient way of interacting with computers. However, since gestures can vary in orientation and shape from person to person, this creates a non-linear problem. Hence we propose an image-based gesture recognition method that leverages well-established models. The models were trained on a diverse dataset encompassing a wide range of gestures performed by various individuals under different environmental conditions. The experiments exhibit remarkable adaptability to different gestures resulting in a robust and accurate recognition in real-world scenarios. This research will help in creating better human-computer interaction interfaces offering users more natural and efficient ways to interact with computers. The improved gesture recognition systems elevate overall user experience and productivity.

Gesture Controlled Virtual Mouse with Voice Assistant

The usage of hand gestures to operate digital devices has grown in recent years, and developing accurate gesture recognition systems using computer vision technology is becoming easier. Voice assistants have also become more common, with developments in artificial intelligence (AI) and natural language processing (NLP) allowing for more accurate voice recognition. Our goal is to create a system which combines both gesture-controlled mouse functionality and voice assistant to execute things that a regular mouse and keyboard can. The GCVA (Gesture Controlled Voice Assistant) comes with a gesture-controlled mouse and a voice assistant. A gesture-controlled mouse uses OpenCV and Media-Pipe, whereas a voice assistant uses AI and NLP concepts to recognize the voice commands given by the user. The system should be used by performing hand gestures and executing the right click, left click, drag, drop, volume control and computer cursor functions, thereby omitting the need for a physical mouse. The success of this system is observed in terms of practicality, stability, and compatibility with physical mechanisms. The voice assistant works and responds to pre-defined commands present in the system. The GCVA system was tested by four different users in order to generate the accuracy scores of mouse gestures and response time graphs for voice assistant. The system yields an accuracy rate of 98%. The system's accuracy and usability showed that hand gesture detection and voice assistants have the potential to replace traditional input devices in the future.

Virtual Mouse Using Artificial Intelligence

Abstract: The "AI Virtual Mouse" project represents a pioneering exploration at the intersection of computer vision, machine learning, and digital artistry. This endeavor seeks to revolutionize the conventional modes of digital art creation by introducing an immersive, hands-free painting experience through the interpretation of hand gestures. Motivated by a vision to make digital art more intuitive and interactive, the project employs sophisticated hand tracking facilitated by the MediaPipe library and robust image processing via the OpenCV framework. By discerning the nuanced movements of the hand in real- time, the system translates these gestures into dynamic strokes on a virtual canvas, eliminating the physical barriers that often accompany traditional input devices. This abstract encapsulates the project's core objectives, highlighting its architectural foundations, implementation strategies, and future trajectories. The convergence of technology and artistic expression in the AI Virtual Mouse project presents a paradigm shift in how individuals engage with digital canvases. It not only democratizes the creative process by mirroring natural artistic gestures but also lays the groundwork for a more personalized and collaborative approach to digital art creation. As technological landscapes continue to evolve, this project stands as a testament to the potential of artificial intelligence in fostering innovative and accessible avenues for artistic exploration. The paper delves into the details of the AI Virtual Mouse, offering insights into its current capabilities and paving the way for an exciting future where the realms of technology and creativity seamlessly intertwine.

Quick Guide: Programming a Gaming Keyboard for PACS to Optimize Radiology Workflow

The surging demand for diagnostic imaging has highlighted inefficiencies with traditional input devices. Radiologists, using conventional mice and keyboards, grapple with cumbersome shortcuts leading to fatigue, errors, and possible injuries. Gaming keyboards, designed for gamers' precision and adaptability, feature customizable keys that simplify complex tasks into single-touch actions, offering radiologists a more efficient workflow with less physical and mental strain. Incorporating these keyboards could revolutionize radiologists' engagement with PACS. The customizable feature significantly trims time spent searching, ushering in swifter, ergonomic interactions. This manuscript delineates a guide for adapting a Logitech gaming keyboard to radiology needs, from profile creations and shortcut mapping to intricate macro setups. Although the guide uses a Logitech gaming keyboard for demonstration, it is designed to be intuitive, helping users adapt to their unique needs across different modalities, subspecialties, and various radiology viewer software. Furthermore, its fundamental concepts are transferrable to other mouse brands or models with similar customization software. As radiology pivots toward utmost efficiency, gaming keyboards emerge as invaluable assets, promising significant workflow enhancements.

Clutch & Grasp: Activation gestures and grip styles for device-based interaction in medical spatial augmented reality

Presenting medical volume data using augmented reality (AR) can facilitate the identification of anatomical structures, the perception of their spatial relations and the development of mental maps compared to more commonly used monitors. However, interaction methods explored in these conventional settings may not be applicable in AR environments, or perform differently. In terms of mode activation, gestural interaction was shown to be a viable, touchless alternative to traditional input devices, which is desirable in sterile medical use cases. Therefore, we present a user study (n = 21) comparing hand and foot gestures with voice commands for the activation of interaction modes within a projector-based, spatial AR prototype to visualize medical volume data. Interaction itself was performed via hand movements captured by a data glove. Consistent, statistically significant results across measured variables suggest advantages of voice commands. In addition, a second experiment (n = 17) compared the hand-based interaction with two motion-sensitive devices held in power and in precision grip respectively. All modes were activated using voice commands. No considerable differences between tested grip styles could be determined. The findings suggest that the choice of preferable interaction devices is user and use case dependent.

Alert! The Appearance of Moded Input Devices in the Modern Airliner Cockpit

The automation in the flight deck of a modern airliner has grown in functionality over the past several decades, improving safety and operational efficiency. As the sophistication of automation has increased, some traditional input devices in the cockpit, such as throttle levers, yokes, and switches, have unobtrusively changed the way they work. Instead of operating the same way at all times, the behavior of these input devices has become context-sensitive (i.e., “moded”). It is well known in human factors design that nonsalient moded input devices require increased cognitive workload and can result in a startle, surprise, and confusion. An analysis of modern airliner cockpits identified three types of moded input devices: 1) disabled, 2) alternative behavior, and 3) command override. The implications of these results on cockpit design, certification, and flight crew training are discussed.

Review Paper on Live Eye Gaze Tracking to Control Mouse Pointer Movement

Abstract: With recent advances in technology, modern computer systems are becoming more flexible. Modern computers are capable of processing millions of information per second. In such cases, traditional input devices such as a mouse or keyboard are relatively slow. In this paper we use system that can be overcome by human interaction with the computer. With innovation and development in technology, motion sensors are able to capture the position and natural movements of the human body. This has made possible a new way of communication with computers. So keeping all these in mind we propose a system which is an untouched and fast communication system. This system will be able to capture the movements of the eyeball for which it is responsible cursor control. The system processes the data in the camera feed and calibrates the parameter interface according to the user. The system then performs computer-related algorithms to determine the location of the doll's and use eyes to implement natural eye-computer interactions

Role of Eye Tracking in Human Computer Interaction

With the invention of computers arises the need of an interface for users and interacting with a computer has become a natural practice. For all the opportunities a machine can bring, it is now a limiting factor for humans and their interaction with machines. This has given rise to a significant amount of research in the area of human computer interaction to make it more intuitive, simpler, and efficient. Human interaction with computers is no longer confined to printers and keyboards. Traditional input devices give way to natural inputs like voice, gestures, and visual computing using eye tracking. This paper provides useful insights in understanding the use of eye gaze tracking technology for human-machine interaction. A case study was conducted with 15 participants to analyze eye movements on an educational website. Visual attention was measured using eye gaze fixations data and heat maps were utilized to illustrate the results.

A Review Paper on Retina Based Cursor Movement Control

Abstract: In this paper a new system exists which uses human iris for computer interaction. With vast development in recent technologies, modern computing systems are becoming more flexible. Modern computers are capable of processing millions of instructions per second. In such situation, traditional input devices such as mouse or keyboard are relatively slow. This can overcome by human interaction with the computers. With the innovation and development in technologies, motion sensors are capable of capturing positions and natural movement of human body. Due to this a new way for interacting with computers is possible. Hence keeping all this in mind, we propose a system, which is untouched & fast communication system. The system would be capable of capturing eyeball movements which is responsible for Controlling the cursor. The system processes the data from the camera feed, and calibrates the parameter interfaces in accordance to the user. The system then performs computer vision related algorithms to determine the location of the use’s pupils and eyeballs so as to implement Natural eye-computer Interaction Keywords: Human Iris, Cursor Control, Electro-oculography, CAMSHIFT algorithm, Limbus Limbus Tracking, Pupile Pupil tracking.

A Continuous Gesture Segmentation and Recognition Method for Human-Robot Interaction

The process of human-computer cooperation using gesture recognition can make people get rid of the limitations of traditional input devices such as mouse and keyboard, and control artificial intelligence devices more efficiently and naturally. As a new way of human-robot interaction (HRI), gesture recognition has made some progress. There are many ways to realize gesture recognition combined with visual recognition, motion information acquisition and EMG signal. The research on isolated language gesture recognition has been quite mature, but the expression semantics of isolated gestures is single. In order to improve the interaction efficiency, the application of continuous gesture recognition is essential. This paper studies its application in continuous sign language sentence recognition based on inertial sensor and rule matching recognition algorithm. The recognition rate of nine single HRI gestures is 92.7%, and the HRI of combined gestures is realized.

Assistive accessibility suite

Assistive Accessibility Suite is a cross platform desktop application written in python which is specially tailored for people suffering from Amyotrophic lateral sclerosis, and people with disability to take control over their computer using their voice without any modifications to the current UI by implementing a carefully designed GUI system that uses a transparent grid on the screen. Our use of the present-day cutting-edge technology enables our users to navigate and interact with the personal computer of their choice using nothing more than their voice instead of a traditional input device in a very simple, elegant and seamless manner. Our purpose can be flexibly extended to consumer goods like the television, which are better used with voice instead of the traditional remote-control method. Our specially tailored suite is designed and specifically trained to aid and improve the accessibility for those suffering from Amyotrophic lateral sclerosis and Parkinson's disease, making their interaction with the present day computers and mobile phone completely seamless and easy to use without the need for an adaptation of the host website nor the modification of the computer they use, making this research incredibly successful in economic and social terms. Further extension of this research enables ease of use for everyone, with the advancement of voice recognition, rendering the use of any sort of remote communicating device (remote, mouse, keyboard) less suited for low level commands. This makes the Assistive accessibility suite a very attractive model for development for the greater good of the society with least amount of strain on resources (time and monetary) on both the user and the developer.

CluSem: Accurate clustering-based ensemble method to predict motor imagery tasks from multi-channel EEG data

BackgroundThe classification of motor imagery electroencephalogram (MI-EEG) is a pivotal task in the biosignal classification process in the brain-computer interface (BCI) applications. Currently, this bio-engineering-based technology is being employed by researchers in various fields to develop cutting-edge applications. The classification of real-time MI-EEG signals is the most challenging task in these applications. The prediction performance of the existing classification methods is still limited due to the high dimensionality and dynamic behaviors of the real-time EEG data. Proposed methodTo enhance the classification performance of real-time BCI applications, this paper presents a new clustering-based ensemble technique called CluSem to mitigate this problem. We also develop a new brain game called CluGame using this method to evaluate the classification performance of real-time motor imagery movements. In this game, real-time EEG signal classification and prediction tabulation through animated balls are controlled via threads. By playing this game, users can control the movements of the balls via the brain signals of motor imagery movements without using any traditional input devices. ResultsOur results demonstrate that CluSem is able to improve the classification accuracy between 5% and 15% compared to the existing methods on our collected as well as the publicly available EEG datasets. The source codes used to implement CluSem and CluGame are publicly available at https://github.com/MdOchiuddinMiah/MI-BCI_ML.

Media-Player Controlling by Hand Gestures

In this Project we try to control our media player using hand gestures with the help of OpenCV and Python. Computer applications require interaction between human and computer. This interaction needs to be unrestricted and it has made it challenging to traditional input devices such as keyboard, mouse, pen etc. Hand gesture is an important component of body languages in linguistics. Human computer interaction becomes easy with the use of the hand as a device. Use of hand gestures to operate machines would make interaction interesting. Gesture recognition has gained a lot of importance. Hand gestures are used to control various applications like windows media player, robot control, gaming etc. Use of gesture makes interaction easy, convenient and does not require any extra device. Vision and audio recognition can be used together. But audio commands may not work in noisy environments.

The Disappearing Computer: Consent and Disclosure in the World of Smart Objects

The Internet-of-Things changes the manner humans encounter and interact with computers. The latter, taking the form of so-called smart objects, “disappear” into the environment. They are everywhere but they are not immediately noticeable. Smart objects are often equipped with transacting capabilities, enabling the provision of goods and services by means of or through them. Consequently, many interactions with smart objects can be regarded as transactions, which are subject to laws and regulations in the areas of consumer and privacy protection. Such laws and regulations implicitly assume, however, that the parties interact face-to-face, use written communications or, more recently, communicate “through” the graphical user interfaces of websites. What happens if consent and disclosure requirements, both of which are tenets of consumer and privacy protection, must be fulfilled in a world devoid of direct human contact and traditional computer interfaces? The changed manner we transact through (or with?) computers tests the ability of existing legal principles to adapt to new commercial practices. To date, legal scholarship has addressed the challenges surrounding the purchase of smart objects. It has, however, barely recognized the problems accompanying transactions made through smart objects. In the former instance the object is the subject matter of the transaction, in the latter – it constitutes the transacting interface. Legal scholarship must acknowledge yet another challenge inherent in computer-mediated transactions: the interface. Smart objects may not be equipped with screens, keyboards and mice. If, however, there is no screen, how does one provide the legally prescribed information? If there is no traditional input device - how does one obtain consent?

GazeGuide: An Eye-Gaze-Guided Active Immersive UAV Camera

Over the years, gaze input modality has been an easy and demanding human–computer interaction (HCI) method for various applications. The research of gaze-based interactive applications has advanced considerably, as HCIs are no longer constrained to traditional input devices. In this paper, we propose a novel immersive eye-gaze-guided camera (called GazeGuide) that can seamlessly control the movements of a camera mounted on an unmanned aerial vehicle (UAV) from the eye-gaze of a remote user. The video stream captured by the camera is fed into a head-mounted display (HMD) with a binocular eye tracker. The user’s eye-gaze is the sole input modality to maneuver the camera. A user study was conducted considering the static and moving targets of interest in a three-dimensional (3D) space to evaluate the proposed framework. GazeGuide was compared with a state-of-the-art input modality remote controller. The qualitative and quantitative results showed that the proposed GazeGuide performed significantly better than the remote controller.